# MediaChunks

chunk_id: 213_chunk_34
chunk_text:
this change did in August of 2019 was make it so that no future roles would become 1780.56s - 1786.24s | misconfigured. Any that had already been created were still vulnerable and that's how we 1786.24s - 1791.24s | were able to assume them in 2024. Later on in July of 2020 variant 1 was introduced to the 1791.60s - 1796.60s | Amplify CLI in studio. And this one was a bit more complicated than the previous one. What 1798.98s - 1804.08s | happened here was that, like I said previously, you can add authentication to an Amplify 1804.08s - 1809.52s | project. If at any time you chose to remove authentication for any reason, that would 1809.52s - 1814.56s | introduce the variant 1 policy. Uh, in speaking to developers who were affected by this, it 1814.56s - 1820.06s | seems the most common reason that this occurred was situations in which uh, uh, projects 1820.08s - 1825.26s | started with the built-in Amplify controlled Cognito resources and then at a later time 1825.26s - 1830.46s | moved to Cognito outside of Amplify or to just a different identity provider entirely. 1830.46s - 1835.50s | Regardless of the reason why, what occurred was that a cloud formation template would run 1835.50s - 1840.00s | that would start setting the default trust policy to what you see on the slide, which is 1840.00s - 1845.78s | totally fine. This is the secure trust policy. However, under the line, uh under the 1845.80s - 1850.38s | condition for the deletion of the authentication configuration, there was a line to delete 1850.38s - 1855.64s | the condition off of the trust policy, which left you with this. The variant 1 vulnerable 1855.64s - 1861.72s | policy. Whatever identity permissions were associated with this rule, when this exposure 1861.72s - 1867.26s | occurred, it still had access to. So if your rule could do something like uh, read from an 1867.26s - 1872.70s | S3 bucket, write to a demo deep table, it still had that access. Additionally, at the end of 1872.72s - 1878.02s | the day, these are just plain IEM rules. And unfortunately, developers love to over 1878.02s - 1883.36s | privilege their identities. When we were reacting to this vulnerability and, and notifying 1883.36s - 1887.80s | customers so they could mitigate it, we found a number of customers


chunk_id: 213_chunk_35
chunk_text:
why, what occurred was that a cloud formation template would run 1835.50s - 1840.00s | that would start setting the default trust policy to what you see on the slide, which is 1840.00s - 1845.78s | totally fine. This is the secure trust policy. However, under the line, uh under the 1845.80s - 1850.38s | condition for the deletion of the authentication configuration, there was a line to delete 1850.38s - 1855.64s | the condition off of the trust policy, which left you with this. The variant 1 vulnerable 1855.64s - 1861.72s | policy. Whatever identity permissions were associated with this rule, when this exposure 1861.72s - 1867.26s | occurred, it still had access to. So if your rule could do something like uh, read from an 1867.26s - 1872.70s | S3 bucket, write to a demo deep table, it still had that access. Additionally, at the end of 1872.72s - 1878.02s | the day, these are just plain IEM rules. And unfortunately, developers love to over 1878.02s - 1883.36s | privilege their identities. When we were reacting to this vulnerability and, and notifying 1883.36s - 1887.80s | customers so they could mitigate it, we found a number of customers who had highly privileged 1887.80s - 1893.58s | roles, including some with S3 full access, Kinesis full access, and more. Unfortunately, as a 1893.58s - 1897.58s | part of this vulnerability, now anyone could have just accessed them and then accessed 1897.60s - 1904.00s | whatever resources were associated with those accounts. Uh, like I said, that was introduced 1904.00s - 1908.84s | in July of 2020, and then between then and when we eventually disclosed in 24, more and more 1908.84s - 1914.02s | roles were becoming misconfigured. One thing I'd really like to stress, uh, that makes this 1914.02s - 1920.42s | cognito issue even more extreme, is that cognito role RNs are easily discoverable. If we 1920.42s - 1924.22s | compare this to the first vulnerability we talked about today, a confused deputy volunteer in 1924.24s - 1930.04s | some AWS service, an adversary still has to contend with the challenge of finding role RNs. To 1930.04s - 1935.08s | be clear, an RN is not a secret, not in the same way a password or API key is, but it does 1935.08s - 1940.06s | represent some level of challenge. They would have to brute force


chunk_id: 213_chunk_36
chunk_text:
who had highly privileged 1887.80s - 1893.58s | roles, including some with S3 full access, Kinesis full access, and more. Unfortunately, as a 1893.58s - 1897.58s | part of this vulnerability, now anyone could have just accessed them and then accessed 1897.60s - 1904.00s | whatever resources were associated with those accounts. Uh, like I said, that was introduced 1904.00s - 1908.84s | in July of 2020, and then between then and when we eventually disclosed in 24, more and more 1908.84s - 1914.02s | roles were becoming misconfigured. One thing I'd really like to stress, uh, that makes this 1914.02s - 1920.42s | cognito issue even more extreme, is that cognito role RNs are easily discoverable. If we 1920.42s - 1924.22s | compare this to the first vulnerability we talked about today, a confused deputy volunteer in 1924.24s - 1930.04s | some AWS service, an adversary still has to contend with the challenge of finding role RNs. To 1930.04s - 1935.08s | be clear, an RN is not a secret, not in the same way a password or API key is, but it does 1935.08s - 1940.06s | represent some level of challenge. They would have to brute force them, enumerate them, drive 1940.06s - 1944.36s | them from IEC, pull them out of build logs, rip them out of GitHub like we did, or a myriad of 1944.36s - 1949.36s | other ways. Suffice to say, it's something you could do across, say, a small number of 1949.40s - 1954.74s | organizations, but not something that you could easily scale to the size of the internet. By 1954.74s - 1959.34s | comparison, the whole point of cognito is that these roles are supposed to be available to your 1959.34s - 1964.58s | users. They're basically public. And in addition, they're also deterministic in the sense that 1964.58s - 1969.28s | if you know the unauth role, you now also know the auth role by default. And so what an 1969.28s - 1974.22s | adversary could have done is simply taken advantage of the default domain associated with 1974.22s - 1980.20s | Amplify apps to suddenly crawl tens of thousands of apps looking for vulnerable roles. And to 1980.20s - 1984.40s | be clear, the majority should be fine, but undoubtedly some number of them were created 1984.40s - 1989.94s | between July, er sorry, uh 2018 and 2019,


chunk_id: 213_chunk_37
chunk_text:
them, enumerate them, drive 1940.06s - 1944.36s | them from IEC, pull them out of build logs, rip them out of GitHub like we did, or a myriad of 1944.36s - 1949.36s | other ways. Suffice to say, it's something you could do across, say, a small number of 1949.40s - 1954.74s | organizations, but not something that you could easily scale to the size of the internet. By 1954.74s - 1959.34s | comparison, the whole point of cognito is that these roles are supposed to be available to your 1959.34s - 1964.58s | users. They're basically public. And in addition, they're also deterministic in the sense that 1964.58s - 1969.28s | if you know the unauth role, you now also know the auth role by default. And so what an 1969.28s - 1974.22s | adversary could have done is simply taken advantage of the default domain associated with 1974.22s - 1980.20s | Amplify apps to suddenly crawl tens of thousands of apps looking for vulnerable roles. And to 1980.20s - 1984.40s | be clear, the majority should be fine, but undoubtedly some number of them were created 1984.40s - 1989.94s | between July, er sorry, uh 2018 and 2019, or who have since changed their authentication 1989.94s - 1994.94s | configuration since July of 2020. Like I said before, we did disclose this to AWS in January 1997.04s - 2002.38s | of this year, and AWS released comprehensive fixes in February and April. Uh they did the 2002.38s - 2006.32s | same thing they did with GitHub Actions, where you can no longer create these vulnerable trust 2006.32s - 2011.16s | policies anymore, but in addition they sort of went above and beyond. Specific to this 2011.16s - 2016.86s | Amplify issue, they actually made changes to STS so that you can no longer assume a role 2016.86s - 2022.90s | cross-account in AWS. So even if you had a variant one or variant two role in your account, 2022.90s - 2027.64s | somebody from a different account can no longer assume it. If you'd like to learn more about 2027.64s - 2032.72s | this vulnerability, AWS did release a security bulletin for it, and we have a blog post here 2032.72s - 2036.72s | with a ton of additional content and information on carrying out these types of attacks, how we 2036.72s - 2042.16s | found this vulnerability,


chunk_id: 213_chunk_38
chunk_text:
or who have since changed their authentication 1989.94s - 1994.94s | configuration since July of 2020. Like I said before, we did disclose this to AWS in January 1997.04s - 2002.38s | of this year, and AWS released comprehensive fixes in February and April. Uh they did the 2002.38s - 2006.32s | same thing they did with GitHub Actions, where you can no longer create these vulnerable trust 2006.32s - 2011.16s | policies anymore, but in addition they sort of went above and beyond. Specific to this 2011.16s - 2016.86s | Amplify issue, they actually made changes to STS so that you can no longer assume a role 2016.86s - 2022.90s | cross-account in AWS. So even if you had a variant one or variant two role in your account, 2022.90s - 2027.64s | somebody from a different account can no longer assume it. If you'd like to learn more about 2027.64s - 2032.72s | this vulnerability, AWS did release a security bulletin for it, and we have a blog post here 2032.72s - 2036.72s | with a ton of additional content and information on carrying out these types of attacks, how we 2036.72s - 2042.16s | found this vulnerability, and more advice on dealing with assume role with web identity. 2042.16s - 2047.60s | Really quickly though, I did want to mention, um throughout this talk we've been focused on 2047.60s - 2052.94s | creating resources in an attacker account so that we could then victimize a victim account. 2052.94s - 2057.94s | The reason for doing this is simplicity. If you own the resource, you can configure it, you can 2057.94s - 2062.48s | set it up however you like, you know it'll always be available, you can troubleshoot it, but I 2062.48s - 2068.12s | wanted to mention it's not strictly required, it's just easier. Specific to the Amplify 2068.12s - 2073.72s | vulnerability, they fixed cross-account role assumption using identity pools. It's no longer 2073.72s - 2079.24s | possible for me to use one of my identity pools to assume a role in your account. But they 2079.26s - 2083.14s | didn't do anything about the same account scenario. And to be clear, I don't really think 2083.14s - 2088.40s | there's much they could do. Uh the reason they were able to block cross-account assume role 2088.40s - 2093.42s | using this is, well that's,


chunk_id: 213_chunk_39
chunk_text:
and more advice on dealing with assume role with web identity. 2042.16s - 2047.60s | Really quickly though, I did want to mention, um throughout this talk we've been focused on 2047.60s - 2052.94s | creating resources in an attacker account so that we could then victimize a victim account. 2052.94s - 2057.94s | The reason for doing this is simplicity. If you own the resource, you can configure it, you can 2057.94s - 2062.48s | set it up however you like, you know it'll always be available, you can troubleshoot it, but I 2062.48s - 2068.12s | wanted to mention it's not strictly required, it's just easier. Specific to the Amplify 2068.12s - 2073.72s | vulnerability, they fixed cross-account role assumption using identity pools. It's no longer 2073.72s - 2079.24s | possible for me to use one of my identity pools to assume a role in your account. But they 2079.26s - 2083.14s | didn't do anything about the same account scenario. And to be clear, I don't really think 2083.14s - 2088.40s | there's much they could do. Uh the reason they were able to block cross-account assume role 2088.40s - 2093.42s | using this is, well that's, it's a very weird thing to do. It's very likely zero if any real 2095.62s - 2100.12s | organizations were doing this in the wild. And so they said hey, we can just block this off, we 2100.12s - 2105.10s | can fix this attack vector and we'll mitigate a lot of the risk. But you can still absolutely 2105.12s - 2110.46s | do this in the same account and this is something that you can still take advantage of today. 2110.46s - 2115.16s | Um to be clear, this is more difficult, things are much nicer if you can just bring your own 2115.16s - 2119.44s | identity pool, but it's definitely something to be aware of if you are penetration duster, red 2119.44s - 2124.94s | teamer, or you simply use Amplify a lot. The way this would work is that if there is a variant 2124.94s - 2131.48s | one or variant two role in that account, if there is also a identity pool that is configured 2131.50s - 2137.44s | with basic authentication or the basic offload, not necessarily associated with Amplify, just 2137.44s - 2143.48s | any Cognito identity pool with that enabled, you as


chunk_id: 213_chunk_40
chunk_text:
it's a very weird thing to do. It's very likely zero if any real 2095.62s - 2100.12s | organizations were doing this in the wild. And so they said hey, we can just block this off, we 2100.12s - 2105.10s | can fix this attack vector and we'll mitigate a lot of the risk. But you can still absolutely 2105.12s - 2110.46s | do this in the same account and this is something that you can still take advantage of today. 2110.46s - 2115.16s | Um to be clear, this is more difficult, things are much nicer if you can just bring your own 2115.16s - 2119.44s | identity pool, but it's definitely something to be aware of if you are penetration duster, red 2119.44s - 2124.94s | teamer, or you simply use Amplify a lot. The way this would work is that if there is a variant 2124.94s - 2131.48s | one or variant two role in that account, if there is also a identity pool that is configured 2131.50s - 2137.44s | with basic authentication or the basic offload, not necessarily associated with Amplify, just 2137.44s - 2143.48s | any Cognito identity pool with that enabled, you as an external adversary can still interact 2143.48s - 2148.58s | with that identity pool, generate a JWT, pass it into Sumo with WebIdentity, and still assume 2148.58s - 2153.78s | one of those variant one or variant two vulnerable roles. Um in the bottom left hand side 2153.78s - 2157.52s | there's a QR code that'll take you to a Hacking the Cloud article that will walk you through 2157.54s - 2164.24s | step by step how you can do this in both authenticated and unauthenticated environments. 2164.24s - 2168.34s | With the time we have left, I wanted to talk a little bit about what we can do to prevent 2168.34s - 2174.34s | cross-tenant attacks in AWS. To be clear, uh your time is probably much better spent doing 2174.34s - 2178.92s | things like deleting old access keys, uh cleaning up old IAM users, following least 2178.92s - 2183.72s | privilege and so on, but if you're already doing all those things, or you have the technology 2183.72s - 2187.70s | and processes in place, it might be helpful to know that we do have some controls that can 2187.70s - 2193.50s | potentially prevent some of


chunk_id: 213_chunk_41
chunk_text:
an external adversary can still interact 2143.48s - 2148.58s | with that identity pool, generate a JWT, pass it into Sumo with WebIdentity, and still assume 2148.58s - 2153.78s | one of those variant one or variant two vulnerable roles. Um in the bottom left hand side 2153.78s - 2157.52s | there's a QR code that'll take you to a Hacking the Cloud article that will walk you through 2157.54s - 2164.24s | step by step how you can do this in both authenticated and unauthenticated environments. 2164.24s - 2168.34s | With the time we have left, I wanted to talk a little bit about what we can do to prevent 2168.34s - 2174.34s | cross-tenant attacks in AWS. To be clear, uh your time is probably much better spent doing 2174.34s - 2178.92s | things like deleting old access keys, uh cleaning up old IAM users, following least 2178.92s - 2183.72s | privilege and so on, but if you're already doing all those things, or you have the technology 2183.72s - 2187.70s | and processes in place, it might be helpful to know that we do have some controls that can 2187.70s - 2193.50s | potentially prevent some of these attacks. In particular, AWS offers us a number of 2193.50s - 2199.70s | condition keys that allow us to restrict essentially where an adversary is coming from. Sort 2199.70s - 2205.04s | of similar to what we saw with a Sumo with WebIdentity previously. These conditions allow us 2205.04s - 2210.42s | to restrict either the ARN, the account, or the organization that is permitted to assume a 2210.42s - 2215.22s | role. It's really important to stress, these are not included by default, uh if you were to 2215.22s - 2219.06s | try to create an instance profile today, it wouldn't automatically add one of these, so you 2219.06s - 2223.36s | would have to manually attach these conditions to each role you want to assume, or want to 2223.36s - 2228.16s | protect, and more importantly, make sure that those services are compatible with these 2228.16s - 2234.04s | condition keys. From some nominal testing, EC2, Lambda, AppSync, all work totally fine, but 2234.04s - 2238.44s | again, really encourage you to test before you try pushing this into production. To 2238.46s - 2243.64s | demonstrate how this would work in practice, uh using source account as


chunk_id: 213_chunk_42
chunk_text:
these attacks. In particular, AWS offers us a number of 2193.50s - 2199.70s | condition keys that allow us to restrict essentially where an adversary is coming from. Sort 2199.70s - 2205.04s | of similar to what we saw with a Sumo with WebIdentity previously. These conditions allow us 2205.04s - 2210.42s | to restrict either the ARN, the account, or the organization that is permitted to assume a 2210.42s - 2215.22s | role. It's really important to stress, these are not included by default, uh if you were to 2215.22s - 2219.06s | try to create an instance profile today, it wouldn't automatically add one of these, so you 2219.06s - 2223.36s | would have to manually attach these conditions to each role you want to assume, or want to 2223.36s - 2228.16s | protect, and more importantly, make sure that those services are compatible with these 2228.16s - 2234.04s | condition keys. From some nominal testing, EC2, Lambda, AppSync, all work totally fine, but 2234.04s - 2238.44s | again, really encourage you to test before you try pushing this into production. To 2238.46s - 2243.64s | demonstrate how this would work in practice, uh using source account as an example and 2243.64s - 2248.88s | tying us back to the first uh confused deputy attack we had in AWS, uh by exploiting this 2248.88s - 2253.52s | vulnerability, an adversary would be able to assume a role which trusted the AppSync service. 2253.52s - 2259.32s | But if we had set a condition key on that trust policy, limiting what AWS account is 2259.32s - 2264.56s | permitted to assume that role, it would successfully block that adversary. The way we 2264.58s - 2268.42s | know this is back in the day when this vulnerability was present, we were able to actually 2268.42s - 2273.52s | test setting that condition key, and it did successfully block an external adversary. So 2273.52s - 2278.66s | even in that worst case scenario of someone having a cross-tenant zero day in AWS, by 2278.66s - 2284.60s | setting this condition key, we can block that access. The other reason I like this is from a 2284.60s - 2290.04s | defense in depth perspective. If pass rule says, under absolutely no circumstance should 2290.06s - 2294.60s | someone be able to assume a role cross-tenant, well great, we broke that with SpongeBob


chunk_id: 213_chunk_43
chunk_text:
an example and 2243.64s - 2248.88s | tying us back to the first uh confused deputy attack we had in AWS, uh by exploiting this 2248.88s - 2253.52s | vulnerability, an adversary would be able to assume a role which trusted the AppSync service. 2253.52s - 2259.32s | But if we had set a condition key on that trust policy, limiting what AWS account is 2259.32s - 2264.56s | permitted to assume that role, it would successfully block that adversary. The way we 2264.58s - 2268.42s | know this is back in the day when this vulnerability was present, we were able to actually 2268.42s - 2273.52s | test setting that condition key, and it did successfully block an external adversary. So 2273.52s - 2278.66s | even in that worst case scenario of someone having a cross-tenant zero day in AWS, by 2278.66s - 2284.60s | setting this condition key, we can block that access. The other reason I like this is from a 2284.60s - 2290.04s | defense in depth perspective. If pass rule says, under absolutely no circumstance should 2290.06s - 2294.60s | someone be able to assume a role cross-tenant, well great, we broke that with SpongeBob 2294.60s - 2299.36s | memes. So the next time that happens, or if somebody finds a more inventive way around it, 2299.36s - 2304.80s | let's have an additional layer of protection to block these types of attacks. You're also 2304.80s - 2310.64s | unlikely to run into many issues, if it works, it's very unlikely to cause uh problems, it's 2310.64s - 2315.88s | this is more of a static thing that stays the same over time. So in conclusion, we took a 2315.92s - 2320.86s | look today at how confused epi attacks can weaponize cloud services against us. We took a 2320.86s - 2325.90s | look at a very dangerous misconfiguration when using assume role with web identity. And 2325.90s - 2330.16s | again, I strongly encourage you to audit your roles that use it, because if you make this 2330.16s - 2334.28s | mistake, theoretically anybody can assume roles in your account. It's an easy mistake to 2334.28s - 2339.88s | make. AWS actually made it twice. Um and then we also took a look today at how we can use 2339.88s - 2345.38s | uh condition keys to block confused epi attacks in AWS. If


chunk_id: 213_chunk_44
chunk_text:
2294.60s - 2299.36s | memes. So the next time that happens, or if somebody finds a more inventive way around it, 2299.36s - 2304.80s | let's have an additional layer of protection to block these types of attacks. You're also 2304.80s - 2310.64s | unlikely to run into many issues, if it works, it's very unlikely to cause uh problems, it's 2310.64s - 2315.88s | this is more of a static thing that stays the same over time. So in conclusion, we took a 2315.92s - 2320.86s | look today at how confused epi attacks can weaponize cloud services against us. We took a 2320.86s - 2325.90s | look at a very dangerous misconfiguration when using assume role with web identity. And 2325.90s - 2330.16s | again, I strongly encourage you to audit your roles that use it, because if you make this 2330.16s - 2334.28s | mistake, theoretically anybody can assume roles in your account. It's an easy mistake to 2334.28s - 2339.88s | make. AWS actually made it twice. Um and then we also took a look today at how we can use 2339.88s - 2345.38s | uh condition keys to block confused epi attacks in AWS. If you enjoyed this talk, you can 2345.38s - 2349.98s | find more of our research at securitylabs.daydoghq.com. If you have any questions, feel free 2349.98s - 2354.32s | to find me after or reach out to me on social media. Again, my name's Nick Frechette. Thank 2354.32s - 2355.00s | you all so much.


chunk_id: 213_chunk_45
chunk_text:
you enjoyed this talk, you can 2345.38s - 2349.98s | find more of our research at securitylabs.daydoghq.com. If you have any questions, feel free 2349.98s - 2354.32s | to find me after or reach out to me on social media. Again, my name's Nick Frechette. Thank 2354.32s - 2355.00s | you all so much.


chunk_id: 214_chunk_0
chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=n8w85LpRplY", "title": "DEF CON 32 - Measuring the Tor Network - Silvia Puglisi, Roger Dingledine", "description": "Millions of people around the world use Tor every day to protect themselves from surveillance and censorship. While the Tor Browser and its protocol are widely known, the backbone of the Tor ecosystem, its extensive network of volunteer relays, is often subject to speculation and misinformation. The Tor Project is dedicated to supporting this network and fostering a vibrant, diverse community of relay operators.\n\nThis talk will focus on our efforts to maintain a healthy network and community, and detect and mitigate attacks -- all with the help of metrics and analysis of usage patterns. By illustrating how we collect safe-enough metrics for an anonymity network, we will offer insights into how we identify unusual activity and other noteworthy events on the network. We will also discuss our ongoing strategies for addressing current and future network health challenges.\n\nIf you are interested in understanding the inner workings of the Tor network and its relay community and how we keep this vital ecosystem running, this talk is for you.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 2495, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.88s - 10.22s | This text was transcribed using whisper model: large-v2 Okay, wow, this microphone is really loud. 10.22s - 11.06s | Awesome. 11.06s - 13.66s | Welcome everybody to another Tor Talk. 13.66s - 16.10s | I'm Roger, this is Sylvia, 16.10s - 19.14s | and we're gonna tell you about the Tor metrics, 19.14s - 21.24s | how to understand what's going on in the network, 21.24s - 24.92s | and in general, how to grow the community that we're going. 24.92s - 27.58s | So part of what I'm trying to do here, 27.58s - 30.74s | many of you have seen only me as a Tor speaker. 30.74s - 32.82s | I'd like to show you that there are other Tor people 32.82s - 34.74s | in the world who actually do things, 34.74s - 37.46s | and eventually there will be more voices of Tor 37.46s - 43.23s | than just me, and maybe eventually, not even me. 43.23s - 47.19s | Okay, so some motivation, some things to think about. 47.19s - 51.57s | Here is a fellow who was elected in Iran back in 2009, 51.57s - 53.23s | but he ended up not in charge, 53.23s


chunk_id: 214_chunk_1
chunk_text:
text was transcribed using whisper model: large-v2 Okay, wow, this microphone is really loud. 10.22s - 11.06s | Awesome. 11.06s - 13.66s | Welcome everybody to another Tor Talk. 13.66s - 16.10s | I'm Roger, this is Sylvia, 16.10s - 19.14s | and we're gonna tell you about the Tor metrics, 19.14s - 21.24s | how to understand what's going on in the network, 21.24s - 24.92s | and in general, how to grow the community that we're going. 24.92s - 27.58s | So part of what I'm trying to do here, 27.58s - 30.74s | many of you have seen only me as a Tor speaker. 30.74s - 32.82s | I'd like to show you that there are other Tor people 32.82s - 34.74s | in the world who actually do things, 34.74s - 37.46s | and eventually there will be more voices of Tor 37.46s - 43.23s | than just me, and maybe eventually, not even me. 43.23s - 47.19s | Okay, so some motivation, some things to think about. 47.19s - 51.57s | Here is a fellow who was elected in Iran back in 2009, 51.57s - 53.23s | but he ended up not in charge, 53.23s - 56.97s | and many of you might remember the Green Party movement. 56.97s - 59.07s | So long ago, I was talking to a friend 59.07s - 62.83s | who was doing trainings of how to be safe in Iran, 62.83s - 64.23s | how to get around censorship, 64.23s - 67.51s | and he was teaching people open proxies, VPNs, 67.51s - 72.35s | Tor, Syphon, other tools, and then he realized, 72.35s - 75.43s | everybody that I've taught anything other than Tor to 75.43s - 79.35s | is now dead or in jail, so I'm only gonna teach Tor, 79.35s - 82.11s | and that was an eye-opening story when I heard it, 82.11s - 84.31s | and there are two lessons there that we take here. 84.31s - 87.27s | One of them is we really need to understand 87.27s - 88.79s | what's going on in the Tor network, 88.79s - 91.55s | so when a lot of users show up from certain countries, 91.55s - 93.07s | we know that it's happening, 93.07s - 95.19s | but also we need to do it safely. 95.19s - 98.19s | We need to be aware


chunk_id: 214_chunk_2
chunk_text:
- 56.97s | and many of you might remember the Green Party movement. 56.97s - 59.07s | So long ago, I was talking to a friend 59.07s - 62.83s | who was doing trainings of how to be safe in Iran, 62.83s - 64.23s | how to get around censorship, 64.23s - 67.51s | and he was teaching people open proxies, VPNs, 67.51s - 72.35s | Tor, Syphon, other tools, and then he realized, 72.35s - 75.43s | everybody that I've taught anything other than Tor to 75.43s - 79.35s | is now dead or in jail, so I'm only gonna teach Tor, 79.35s - 82.11s | and that was an eye-opening story when I heard it, 82.11s - 84.31s | and there are two lessons there that we take here. 84.31s - 87.27s | One of them is we really need to understand 87.27s - 88.79s | what's going on in the Tor network, 88.79s - 91.55s | so when a lot of users show up from certain countries, 91.55s - 93.07s | we know that it's happening, 93.07s - 95.19s | but also we need to do it safely. 95.19s - 98.19s | We need to be aware of how we can measure things 98.19s - 103.37s | and understand things without putting users at risk. 103.37s - 105.25s | So we've got four main topics 105.25s - 106.27s | we're gonna talk about today. 106.27s - 108.45s | The first one is introduction to Tor 108.45s - 110.73s | for the people who need a refresher, 110.73s - 113.83s | and then we're gonna talk more about the relay community 113.83s - 117.37s | and who runs the relays and how we grow that as a community, 117.37s - 119.17s | and then we're gonna talk about 119.17s - 121.37s | understanding what's going on with metrics 121.37s - 123.29s | and measurements and datasets, 123.29s - 126.79s | and at the end, we'll ask you to help us save the world. 128.65s - 130.17s | Okay, did I? 130.17s - 131.05s | Yes. 131.05s - 133.85s | Okay, so Tor is a lot of different things. 133.85s - 135.61s | Tor is a nonprofit. 135.61s - 137.37s | It's a free software project. 137.37s - 139.09s | It's a network of volunteers. 139.09s - 142.49s | So raise your hand if any of the following apply


chunk_id: 214_chunk_3
chunk_text:
of how we can measure things 98.19s - 103.37s | and understand things without putting users at risk. 103.37s - 105.25s | So we've got four main topics 105.25s - 106.27s | we're gonna talk about today. 106.27s - 108.45s | The first one is introduction to Tor 108.45s - 110.73s | for the people who need a refresher, 110.73s - 113.83s | and then we're gonna talk more about the relay community 113.83s - 117.37s | and who runs the relays and how we grow that as a community, 117.37s - 119.17s | and then we're gonna talk about 119.17s - 121.37s | understanding what's going on with metrics 121.37s - 123.29s | and measurements and datasets, 123.29s - 126.79s | and at the end, we'll ask you to help us save the world. 128.65s - 130.17s | Okay, did I? 130.17s - 131.05s | Yes. 131.05s - 133.85s | Okay, so Tor is a lot of different things. 133.85s - 135.61s | Tor is a nonprofit. 135.61s - 137.37s | It's a free software project. 137.37s - 139.09s | It's a network of volunteers. 139.09s - 142.49s | So raise your hand if any of the following apply to you. 143.39s - 147.37s | Anybody here run a Tor relay or a bridge or a snowflake? 147.37s - 149.25s | Anybody here help with research 149.25s - 152.69s | or helping in writing research papers? 152.69s - 155.89s | Anybody here use Tor, use Tor browser? 155.89s - 158.65s | Anybody here find themselves explaining to their relatives 158.65s - 162.45s | why I have nothing to hide makes no sense? 162.45s - 164.37s | Excellent, you're all part of Tor. 164.37s - 167.73s | Tor is an ecosystem of people trying to fix the fact 167.73s - 172.97s | that people need privacy and not enough of them have it. 172.97s - 174.93s | Okay, so I use the word anonymity 174.93s - 176.21s | when I'm talking to researchers, 176.21s - 179.73s | but really it comes down to communications metadata, 179.73s - 183.25s | and here's creepy NSA guy that we learned a lot more about 183.25s - 184.53s | from this note in documents, 184.53s - 187.85s | and he actually said we kill people based on metadata. 187.85s - 190.21s | So that's where we're starting from, 190.21s - 194.17s | and the


chunk_id: 214_chunk_4
chunk_text:
to you. 143.39s - 147.37s | Anybody here run a Tor relay or a bridge or a snowflake? 147.37s - 149.25s | Anybody here help with research 149.25s - 152.69s | or helping in writing research papers? 152.69s - 155.89s | Anybody here use Tor, use Tor browser? 155.89s - 158.65s | Anybody here find themselves explaining to their relatives 158.65s - 162.45s | why I have nothing to hide makes no sense? 162.45s - 164.37s | Excellent, you're all part of Tor. 164.37s - 167.73s | Tor is an ecosystem of people trying to fix the fact 167.73s - 172.97s | that people need privacy and not enough of them have it. 172.97s - 174.93s | Okay, so I use the word anonymity 174.93s - 176.21s | when I'm talking to researchers, 176.21s - 179.73s | but really it comes down to communications metadata, 179.73s - 183.25s | and here's creepy NSA guy that we learned a lot more about 183.25s - 184.53s | from this note in documents, 184.53s - 187.85s | and he actually said we kill people based on metadata. 187.85s - 190.21s | So that's where we're starting from, 190.21s - 194.17s | and the goal of Tor is to protect that metadata 194.17s - 198.53s | so that it's hard for people to tell who's doing what. 198.53s - 201.77s | So I only use the word anonymity talking to researchers. 201.77s - 204.13s | When I'm talking to my parents and grandparents, 204.13s - 205.95s | I work on privacy systems, 205.95s - 208.81s | because anonymity, I don't know, that's kind of scary, 208.81s - 210.85s | but privacy is an important value. 210.85s - 212.61s | Everybody should have privacy. 212.61s - 216.21s | And then when I'm talking to corporations, 216.21s - 217.89s | Google, Walmart, and so on, 217.89s - 220.45s | I work on communication security, 220.45s - 222.61s | because privacy is dead, anonymity, 222.61s - 223.75s | why would they want that? 223.75s - 225.97s | But of course they want security. 225.97s - 227.73s | And then when I'm talking to governments, 227.73s - 229.21s | I work on traffic analysis 229.21s - 231.41s | resistant communication networks. 231.41s - 233.01s | And again, it's the same system, 233.01s - 234.49s | the same security properties. 234.49s - 236.59s | The goal is to get


chunk_id: 214_chunk_5
chunk_text:
goal of Tor is to protect that metadata 194.17s - 198.53s | so that it's hard for people to tell who's doing what. 198.53s - 201.77s | So I only use the word anonymity talking to researchers. 201.77s - 204.13s | When I'm talking to my parents and grandparents, 204.13s - 205.95s | I work on privacy systems, 205.95s - 208.81s | because anonymity, I don't know, that's kind of scary, 208.81s - 210.85s | but privacy is an important value. 210.85s - 212.61s | Everybody should have privacy. 212.61s - 216.21s | And then when I'm talking to corporations, 216.21s - 217.89s | Google, Walmart, and so on, 217.89s - 220.45s | I work on communication security, 220.45s - 222.61s | because privacy is dead, anonymity, 222.61s - 223.75s | why would they want that? 223.75s - 225.97s | But of course they want security. 225.97s - 227.73s | And then when I'm talking to governments, 227.73s - 229.21s | I work on traffic analysis 229.21s - 231.41s | resistant communication networks. 231.41s - 233.01s | And again, it's the same system, 233.01s - 234.49s | the same security properties. 234.49s - 236.59s | The goal is to get all these users together 236.59s - 238.97s | so you have your cancer survivors 238.97s - 241.27s | and your Egyptian activists and other groups 241.27s - 243.43s | all inside the same network. 243.43s - 245.47s | And then there's the fourth category 245.47s - 247.57s | that we've been thinking a lot about recently, 247.57s - 249.67s | which is the reachability side, 249.67s - 252.61s | the people in Vietnam who can't get to BBC, 252.61s - 255.31s | or people trying to reach the Tor network 255.31s - 257.35s | even though censors are trying to block it. 257.35s - 260.39s | So the goal here is to get all of these different groups 260.39s - 262.19s | into the same anonymity system 262.19s - 263.67s | so that they can blend together. 263.67s - 266.73s | Otherwise, if you just had an anonymity system 266.73s - 268.63s | for cancer survivors, 268.63s - 271.39s | then people would know why you installed it, 271.39s - 274.64s | why you're using it. 274.64s - 276.40s | Okay, so how do you actually build one of these? 276.40s - 279.00s | How do you build a metadata,


chunk_id: 214_chunk_6
chunk_text:
all these users together 236.59s - 238.97s | so you have your cancer survivors 238.97s - 241.27s | and your Egyptian activists and other groups 241.27s - 243.43s | all inside the same network. 243.43s - 245.47s | And then there's the fourth category 245.47s - 247.57s | that we've been thinking a lot about recently, 247.57s - 249.67s | which is the reachability side, 249.67s - 252.61s | the people in Vietnam who can't get to BBC, 252.61s - 255.31s | or people trying to reach the Tor network 255.31s - 257.35s | even though censors are trying to block it. 257.35s - 260.39s | So the goal here is to get all of these different groups 260.39s - 262.19s | into the same anonymity system 262.19s - 263.67s | so that they can blend together. 263.67s - 266.73s | Otherwise, if you just had an anonymity system 266.73s - 268.63s | for cancer survivors, 268.63s - 271.39s | then people would know why you installed it, 271.39s - 274.64s | why you're using it. 274.64s - 276.40s | Okay, so how do you actually build one of these? 276.40s - 279.00s | How do you build a metadata, 279.00s - 281.80s | communications metadata privacy system? 281.80s - 284.80s | The easy answer is you have some centralized system 284.80s - 287.60s | like a VPN, and all the users connect to it 287.60s - 290.12s | and request some webpage or something. 290.12s - 292.14s | And a bunch of things can go wrong here. 292.14s - 294.20s | The first thing that can go wrong is 295.20s - 298.66s | what if that central point of failure fails? 298.66s - 300.42s | So we hear this all the time. 300.42s - 302.46s | There's some new VPN company that says, 302.46s - 305.98s | we promise we won't look at any of your traffic. 305.98s - 308.18s | Okay, actually, we do look at all of your traffic. 308.18s - 310.50s | We promise we won't log anything. 310.50s - 312.36s | Okay, actually, we do write all of it down. 312.36s - 314.26s | We promise we won't tell anybody. 314.26s - 316.46s | Oh, well, of course we answer subpoenas. 316.46s - 319.66s | And so the problem there is you don't know. 319.66s - 321.14s | It's a centralized system. 321.14s - 323.90s


chunk_id: 214_chunk_7
chunk_text:
279.00s - 281.80s | communications metadata privacy system? 281.80s - 284.80s | The easy answer is you have some centralized system 284.80s - 287.60s | like a VPN, and all the users connect to it 287.60s - 290.12s | and request some webpage or something. 290.12s - 292.14s | And a bunch of things can go wrong here. 292.14s - 294.20s | The first thing that can go wrong is 295.20s - 298.66s | what if that central point of failure fails? 298.66s - 300.42s | So we hear this all the time. 300.42s - 302.46s | There's some new VPN company that says, 302.46s - 305.98s | we promise we won't look at any of your traffic. 305.98s - 308.18s | Okay, actually, we do look at all of your traffic. 308.18s - 310.50s | We promise we won't log anything. 310.50s - 312.36s | Okay, actually, we do write all of it down. 312.36s - 314.26s | We promise we won't tell anybody. 314.26s - 316.46s | Oh, well, of course we answer subpoenas. 316.46s - 319.66s | And so the problem there is you don't know. 319.66s - 321.14s | It's a centralized system. 321.14s - 323.90s | They promise they won't screw you, but they have the data. 323.90s - 326.34s | They know you, they know what you do. 326.34s - 327.94s | It's privacy by promise. 328.92s - 330.18s | And it's worse than that. 331.78s - 334.10s | Even if that centralized provider 334.10s - 336.78s | is somehow magically honest in all ways, 336.78s - 338.50s | it's still one point. 338.50s - 340.94s | And the cable goes in, the same cable comes out. 340.94s - 342.54s | And if you can get access to that, 342.54s - 345.08s | then you can start doing timing and volume analysis 345.08s - 348.42s | to say that user wanted to get that webpage. 348.42s - 351.46s | So even if the central point is totally honest, 351.46s - 352.64s | it's still bad news. 353.86s - 356.62s | So the goal of Tor is to distribute the trust 356.62s - 359.60s | over multiple relays so there's no single point 359.60s - 362.06s | that knows both you and what you're doing. 362.06s - 365.34s | So it's privacy by design, privacy by architecture, 365.34s - 367.42s | rather than privacy by


chunk_id: 214_chunk_8
chunk_text:
| They promise they won't screw you, but they have the data. 323.90s - 326.34s | They know you, they know what you do. 326.34s - 327.94s | It's privacy by promise. 328.92s - 330.18s | And it's worse than that. 331.78s - 334.10s | Even if that centralized provider 334.10s - 336.78s | is somehow magically honest in all ways, 336.78s - 338.50s | it's still one point. 338.50s - 340.94s | And the cable goes in, the same cable comes out. 340.94s - 342.54s | And if you can get access to that, 342.54s - 345.08s | then you can start doing timing and volume analysis 345.08s - 348.42s | to say that user wanted to get that webpage. 348.42s - 351.46s | So even if the central point is totally honest, 351.46s - 352.64s | it's still bad news. 353.86s - 356.62s | So the goal of Tor is to distribute the trust 356.62s - 359.60s | over multiple relays so there's no single point 359.60s - 362.06s | that knows both you and what you're doing. 362.06s - 365.34s | So it's privacy by design, privacy by architecture, 365.34s - 367.42s | rather than privacy by promise. 367.42s - 370.66s | And part of the goal is we've got these, 370.66s - 372.34s | you're able to see the slides, right, 372.34s - 373.28s | even though they're not here? 373.28s - 374.46s | Okay, good. 374.46s - 377.30s | So part of the goal is that we've got these three relays 377.30s - 380.00s | and you build your path through the three relays 380.00s - 382.52s | and no single point in the network 382.52s - 384.40s | is able to learn both who you are 384.40s - 386.34s | and what destinations you're going to. 387.84s - 389.40s | And another key point here, 389.40s - 391.52s | transparency for Tor is really important. 391.52s - 393.80s | Yes, it's open source, it's free software. 393.80s - 395.60s | Yes, we give you the design documents 395.60s - 398.04s | and the specifications and the research papers. 398.04s - 401.96s | And yes, we say, hi, I'm Roger, she's Sylvia. 401.96s - 403.76s | We're people, we're actually developers 403.76s - 407.02s | who are happy to publicly identify ourselves. 407.02s - 410.64s | And the key to that is transparency is really


chunk_id: 214_chunk_9
chunk_text:
promise. 367.42s - 370.66s | And part of the goal is we've got these, 370.66s - 372.34s | you're able to see the slides, right, 372.34s - 373.28s | even though they're not here? 373.28s - 374.46s | Okay, good. 374.46s - 377.30s | So part of the goal is that we've got these three relays 377.30s - 380.00s | and you build your path through the three relays 380.00s - 382.52s | and no single point in the network 382.52s - 384.40s | is able to learn both who you are 384.40s - 386.34s | and what destinations you're going to. 387.84s - 389.40s | And another key point here, 389.40s - 391.52s | transparency for Tor is really important. 391.52s - 393.80s | Yes, it's open source, it's free software. 393.80s - 395.60s | Yes, we give you the design documents 395.60s - 398.04s | and the specifications and the research papers. 398.04s - 401.96s | And yes, we say, hi, I'm Roger, she's Sylvia. 401.96s - 403.76s | We're people, we're actually developers 403.76s - 407.02s | who are happy to publicly identify ourselves. 407.02s - 410.64s | And the key to that is transparency is really important. 410.64s - 412.96s | Every time I talk to somebody, they say, 412.96s - 414.64s | oh, ha ha, the privacy people 414.64s - 415.96s | are talking about transparency. 415.96s - 417.38s | That doesn't make any sense. 417.38s - 421.16s | And the reason why it makes sense is privacy is about choice. 421.16s - 423.04s | Transparency is about choice. 423.04s - 426.52s | Our goal here is we choose to be transparent 426.52s - 428.56s | in order to build a stronger community, 428.56s - 431.80s | in order for people to be able to trust what we're doing 431.80s - 436.60s | and to become part of our community. 436.60s - 438.36s | Okay, so here's our first graph. 438.36s - 440.82s | This is a graph of the capacity and the load 440.82s - 442.96s | on the Tor network over the past 10 years. 443.00s - 446.28s | The green line is what sort of capacity we have, 446.28s - 448.68s | and the purple line is the load on the network. 448.68s - 451.80s | And we're up at 250 gigabits per second 451.80s - 454.28s | of actual load on


chunk_id: 214_chunk_10
chunk_text:
important. 410.64s - 412.96s | Every time I talk to somebody, they say, 412.96s - 414.64s | oh, ha ha, the privacy people 414.64s - 415.96s | are talking about transparency. 415.96s - 417.38s | That doesn't make any sense. 417.38s - 421.16s | And the reason why it makes sense is privacy is about choice. 421.16s - 423.04s | Transparency is about choice. 423.04s - 426.52s | Our goal here is we choose to be transparent 426.52s - 428.56s | in order to build a stronger community, 428.56s - 431.80s | in order for people to be able to trust what we're doing 431.80s - 436.60s | and to become part of our community. 436.60s - 438.36s | Okay, so here's our first graph. 438.36s - 440.82s | This is a graph of the capacity and the load 440.82s - 442.96s | on the Tor network over the past 10 years. 443.00s - 446.28s | The green line is what sort of capacity we have, 446.28s - 448.68s | and the purple line is the load on the network. 448.68s - 451.80s | And we're up at 250 gigabits per second 451.80s - 454.28s | of actual load on the network, 454.28s - 457.52s | which is a pretty big global network at this point. 457.52s - 460.40s | On the other hand, the capacity has still been growing, 460.40s - 462.60s | and the purple line, the load line, 462.60s - 465.08s | has sort of flattened out over the past couple of years. 465.08s - 468.42s | So we can definitely talk about how to scale 468.42s - 477.38s | and how the performance should work. 477.38s - 479.58s | No, keep going. 480.42s - 481.26s | That's fine. 482.50s - 483.78s | Now it works? 483.78s - 484.62s | Oh, it doesn't. 485.58s - 487.10s | Hello, that's fine. 487.10s - 492.10s | Okay, so this is the number of relays per flags that we have. 492.50s - 496.58s | Flags are like roles that relay can have on the network. 496.58s - 499.42s | And for example, if we have a stable relay, 499.42s - 501.82s | that usually is gonna be a guard as well, 501.82s - 504.70s | meaning that that's the relay that the Tor browser picks 504.70s - 506.66s | when entering the network. 506.66s - 508.62s | And some of


chunk_id: 214_chunk_11
chunk_text:
the network, 454.28s - 457.52s | which is a pretty big global network at this point. 457.52s - 460.40s | On the other hand, the capacity has still been growing, 460.40s - 462.60s | and the purple line, the load line, 462.60s - 465.08s | has sort of flattened out over the past couple of years. 465.08s - 468.42s | So we can definitely talk about how to scale 468.42s - 477.38s | and how the performance should work. 477.38s - 479.58s | No, keep going. 480.42s - 481.26s | That's fine. 482.50s - 483.78s | Now it works? 483.78s - 484.62s | Oh, it doesn't. 485.58s - 487.10s | Hello, that's fine. 487.10s - 492.10s | Okay, so this is the number of relays per flags that we have. 492.50s - 496.58s | Flags are like roles that relay can have on the network. 496.58s - 499.42s | And for example, if we have a stable relay, 499.42s - 501.82s | that usually is gonna be a guard as well, 501.82s - 504.70s | meaning that that's the relay that the Tor browser picks 504.70s - 506.66s | when entering the network. 506.66s - 508.62s | And some of you might notice 508.62s - 510.70s | there is an interesting curve here. 510.70s - 514.66s | It's the HSDR graph here. 514.66s - 516.88s | In the middle, we have 4,000 relays, 516.88s - 519.26s | and then sometimes around June, something happens, 519.26s - 522.06s | and we lose 2,000 of those relays. 522.06s - 524.30s | And why does that happen? 524.30s - 527.30s | So there was a Debian update. 527.30s - 529.46s | There was a library that had a few dependencies 529.46s - 530.74s | in common with Tor. 530.74s - 534.02s | So that updated, and the Tor daemon restarted, 534.02s - 536.42s | and the relays lose the flag. 536.42s - 538.70s | And this introduced us to a topic 538.70s - 542.02s | that is important for the Tor network, which is diversity. 542.02s - 544.94s | And actually shows that we have a little bit 544.94s - 548.62s | of a monoculture issue when it comes to Tor relays. 548.62s - 551.70s | A lot of them don't run Debian. 551.70s - 554.46s | And that is because the easiest way to run a Tor relay 554.46s - 558.18s |


chunk_id: 214_chunk_12
chunk_text:
you might notice 508.62s - 510.70s | there is an interesting curve here. 510.70s - 514.66s | It's the HSDR graph here. 514.66s - 516.88s | In the middle, we have 4,000 relays, 516.88s - 519.26s | and then sometimes around June, something happens, 519.26s - 522.06s | and we lose 2,000 of those relays. 522.06s - 524.30s | And why does that happen? 524.30s - 527.30s | So there was a Debian update. 527.30s - 529.46s | There was a library that had a few dependencies 529.46s - 530.74s | in common with Tor. 530.74s - 534.02s | So that updated, and the Tor daemon restarted, 534.02s - 536.42s | and the relays lose the flag. 536.42s - 538.70s | And this introduced us to a topic 538.70s - 542.02s | that is important for the Tor network, which is diversity. 542.02s - 544.94s | And actually shows that we have a little bit 544.94s - 548.62s | of a monoculture issue when it comes to Tor relays. 548.62s - 551.70s | A lot of them don't run Debian. 551.70s - 554.46s | And that is because the easiest way to run a Tor relay 554.46s - 558.18s | is you spin up a Debian machine, 558.18s - 560.90s | you install the package, and then you have the relay. 562.02s - 566.02s | And so it shows these monoculture issues, 566.66s - 568.30s | but I think we're not the only one 568.30s - 569.58s | to have monoculture issues. 569.58s - 574.14s | I think Microsoft said that they had 8.5 million devices 574.14s - 578.86s | affected by the CrowdStrike incident, so anyway. 580.30s - 582.74s | And this is why we talk about diversity a lot 582.74s - 585.70s | when we talk about the Tor network. 585.70s - 588.46s | The Tor network is driven by a diverse community 588.46s - 593.46s | of relay operators, and we have developers, 593.62s - 597.06s | we have privacy researchers, we have security professional, 597.06s - 598.46s | we have academics. 599.58s - 602.54s | And diversity is not just a desirable attribute, 602.54s - 603.98s | but it's a critical component 603.98s - 605.94s | when we talk about the network. 605.94s - 610.10s | And you might wonder, what do we mean by diversity? 610.10s - 612.38s | Why we place so much emphasis


chunk_id: 214_chunk_13
chunk_text:
is you spin up a Debian machine, 558.18s - 560.90s | you install the package, and then you have the relay. 562.02s - 566.02s | And so it shows these monoculture issues, 566.66s - 568.30s | but I think we're not the only one 568.30s - 569.58s | to have monoculture issues. 569.58s - 574.14s | I think Microsoft said that they had 8.5 million devices 574.14s - 578.86s | affected by the CrowdStrike incident, so anyway. 580.30s - 582.74s | And this is why we talk about diversity a lot 582.74s - 585.70s | when we talk about the Tor network. 585.70s - 588.46s | The Tor network is driven by a diverse community 588.46s - 593.46s | of relay operators, and we have developers, 593.62s - 597.06s | we have privacy researchers, we have security professional, 597.06s - 598.46s | we have academics. 599.58s - 602.54s | And diversity is not just a desirable attribute, 602.54s - 603.98s | but it's a critical component 603.98s - 605.94s | when we talk about the network. 605.94s - 610.10s | And you might wonder, what do we mean by diversity? 610.10s - 612.38s | Why we place so much emphasis on it? 613.70s - 618.18s | And what mechanism, and we do have to foster 618.18s - 619.46s | this diverse community. 619.46s - 623.34s | So the first thing is when we talk about the network, 624.22s - 625.86s | we mean diversity of hardware, 625.86s - 629.18s | software configuration, location, but also of users. 630.26s - 631.98s | Hardware and software configuration, 631.98s - 634.38s | we saw a little bit when a lot of relays 634.38s - 636.02s | run the same operating system, 636.02s - 637.82s | they are affected maybe by a bug. 637.82s - 639.78s | That means a big part of the network 639.78s - 641.90s | is affected by the same bug. 641.90s - 645.70s | So we strive, our objective is to have 645.70s - 648.94s | a uniform distribution of operating systems. 648.94s - 651.70s | Of course, we're not there, but that's what we aim for. 652.66s - 655.74s | We place a lot of emphasis on location as well 655.74s - 657.34s | for a variety of reasons. 657.34s - 660.38s | We want third circuit to spread 660.38s - 662.02s | different geographical location 662.02s -


chunk_id: 214_chunk_14
chunk_text:
on it? 613.70s - 618.18s | And what mechanism, and we do have to foster 618.18s - 619.46s | this diverse community. 619.46s - 623.34s | So the first thing is when we talk about the network, 624.22s - 625.86s | we mean diversity of hardware, 625.86s - 629.18s | software configuration, location, but also of users. 630.26s - 631.98s | Hardware and software configuration, 631.98s - 634.38s | we saw a little bit when a lot of relays 634.38s - 636.02s | run the same operating system, 636.02s - 637.82s | they are affected maybe by a bug. 637.82s - 639.78s | That means a big part of the network 639.78s - 641.90s | is affected by the same bug. 641.90s - 645.70s | So we strive, our objective is to have 645.70s - 648.94s | a uniform distribution of operating systems. 648.94s - 651.70s | Of course, we're not there, but that's what we aim for. 652.66s - 655.74s | We place a lot of emphasis on location as well 655.74s - 657.34s | for a variety of reasons. 657.34s - 660.38s | We want third circuit to spread 660.38s - 662.02s | different geographical location 662.02s - 665.58s | because it would make harder for an observer 665.58s - 670.58s | to place the origin and the destination of a communication. 671.22s - 674.58s | We want also to our relays to be spread globally 674.58s - 676.46s | to avoid single point of failure. 677.78s - 679.98s | Because if there is an issues locally 680.02s - 682.78s | that is affecting a number of relays, 682.78s - 684.74s | we know that the rest of the network, 684.74s - 686.86s | we maintain the stability. 686.86s - 690.26s | And we also think that by having relay operators 690.26s - 691.46s | spread around the world, 691.46s - 694.98s | we have a more diverse community, diverse location, 694.98s - 699.10s | which in the end will help maintain the network growth 699.10s - 701.90s | and sustainability in the long run. 701.90s - 705.34s | And we also want diversity of users. 705.34s - 708.94s | So people use Tor for a variety of reasons. 708.94s - 712.58s | We estimate that we have 400,000 users 712.58s - 714.82s | daily in the United States. 714.82s - 718.74s | It means that we have


chunk_id: 214_chunk_15
chunk_text:
665.58s | because it would make harder for an observer 665.58s - 670.58s | to place the origin and the destination of a communication. 671.22s - 674.58s | We want also to our relays to be spread globally 674.58s - 676.46s | to avoid single point of failure. 677.78s - 679.98s | Because if there is an issues locally 680.02s - 682.78s | that is affecting a number of relays, 682.78s - 684.74s | we know that the rest of the network, 684.74s - 686.86s | we maintain the stability. 686.86s - 690.26s | And we also think that by having relay operators 690.26s - 691.46s | spread around the world, 691.46s - 694.98s | we have a more diverse community, diverse location, 694.98s - 699.10s | which in the end will help maintain the network growth 699.10s - 701.90s | and sustainability in the long run. 701.90s - 705.34s | And we also want diversity of users. 705.34s - 708.94s | So people use Tor for a variety of reasons. 708.94s - 712.58s | We estimate that we have 400,000 users 712.58s - 714.82s | daily in the United States. 714.82s - 718.74s | It means that we have diversity of usage patterns, 718.74s - 720.86s | profiles, interest. 720.86s - 725.02s | We have minority groups using Tor, social right activists. 725.02s - 728.26s | We have privacy enthusiasts who would want to be followed 728.26s - 731.58s | by surveillance company, why they surf the web. 731.58s - 736.10s | And for many people like activists, 737.10s - 740.70s | people belonging to different kind, 740.70s - 742.50s | for example, people researching, 746.07s - 747.11s | people researching, 748.99s - 751.79s | people researching like information 751.79s - 753.51s | on reproductive rights online. 753.51s - 757.19s | People needed looking for information 757.19s - 758.23s | on domestic violence. 758.23s - 762.47s | For example, Tor provides a platform to access information 762.47s - 764.83s | and to express themselves in environments 764.87s - 767.55s | where they could otherwise face persecution, 767.55s - 769.99s | discrimination, and retaliation. 771.63s - 776.63s | So how do we actually foster this diverse community 777.15s - 777.99s | in a healthy way? 777.99s - 781.55s | So we place a lot of emphasis on direct communication. 781.55s - 785.55s | We use IRC, we use mailing lists,


chunk_id: 214_chunk_16
chunk_text:
diversity of usage patterns, 718.74s - 720.86s | profiles, interest. 720.86s - 725.02s | We have minority groups using Tor, social right activists. 725.02s - 728.26s | We have privacy enthusiasts who would want to be followed 728.26s - 731.58s | by surveillance company, why they surf the web. 731.58s - 736.10s | And for many people like activists, 737.10s - 740.70s | people belonging to different kind, 740.70s - 742.50s | for example, people researching, 746.07s - 747.11s | people researching, 748.99s - 751.79s | people researching like information 751.79s - 753.51s | on reproductive rights online. 753.51s - 757.19s | People needed looking for information 757.19s - 758.23s | on domestic violence. 758.23s - 762.47s | For example, Tor provides a platform to access information 762.47s - 764.83s | and to express themselves in environments 764.87s - 767.55s | where they could otherwise face persecution, 767.55s - 769.99s | discrimination, and retaliation. 771.63s - 776.63s | So how do we actually foster this diverse community 777.15s - 777.99s | in a healthy way? 777.99s - 781.55s | So we place a lot of emphasis on direct communication. 781.55s - 785.55s | We use IRC, we use mailing lists, we use forum. 785.55s - 788.27s | We organize online and in-person meetups, 788.27s - 791.67s | online meetups for our community up and every month. 791.67s - 794.99s | We get feedback from relay operators, for example, 794.99s - 799.47s | and we tell them how development is going, 799.47s - 801.27s | if there is any update, 801.27s - 804.23s | and we get this communication going on. 804.23s - 808.15s | We have in-person meetups, we are here at DEF CON. 808.15s - 810.15s | In Europe, we know that there are other events 810.15s - 812.27s | where people in our community like to gather. 812.27s - 815.91s | So for example, we have FOSDEM, we have CCC. 815.91s - 818.67s | And we do develop transparent policies. 819.67s - 823.43s | We have open process to create and maintain policies 823.43s - 824.67s | for our community, 824.67s - 827.11s | which are central to how we run the community. 828.31s - 832.19s | This discussion of how to propose and maintain this policy 832.19s - 835.39s | happen on our backtracker on GitLab. 835.39s - 839.15s | And everybody's welcome to give feedback. 839.15s


chunk_id: 214_chunk_17
chunk_text:
we use forum. 785.55s - 788.27s | We organize online and in-person meetups, 788.27s - 791.67s | online meetups for our community up and every month. 791.67s - 794.99s | We get feedback from relay operators, for example, 794.99s - 799.47s | and we tell them how development is going, 799.47s - 801.27s | if there is any update, 801.27s - 804.23s | and we get this communication going on. 804.23s - 808.15s | We have in-person meetups, we are here at DEF CON. 808.15s - 810.15s | In Europe, we know that there are other events 810.15s - 812.27s | where people in our community like to gather. 812.27s - 815.91s | So for example, we have FOSDEM, we have CCC. 815.91s - 818.67s | And we do develop transparent policies. 819.67s - 823.43s | We have open process to create and maintain policies 823.43s - 824.67s | for our community, 824.67s - 827.11s | which are central to how we run the community. 828.31s - 832.19s | This discussion of how to propose and maintain this policy 832.19s - 835.39s | happen on our backtracker on GitLab. 835.39s - 839.15s | And everybody's welcome to give feedback. 839.15s - 842.39s | And it's a way for us to give new tools 842.39s - 843.51s | to the relay community, 843.51s - 844.55s | because in the past, 844.55s - 847.67s | we noticed that they were organizing informally. 847.67s - 849.67s | And this is a tool to help them 849.67s - 854.67s | with managing expectation and with governance. 860.56s - 862.00s | Testing, testing, great. 862.00s - 864.40s | So where do these relays come from? 864.40s - 867.24s | And originally, it was people that I had met 867.24s - 869.16s | at hacker conferences like this. 869.16s - 871.00s | And I said, tour is really important. 871.00s - 872.72s | And you said, how can I help? 872.72s - 874.64s | And I convinced you to run a relay. 874.64s - 877.76s | And that worked great for the first 50 relays 877.76s - 879.00s | or 100 relays. 879.04s - 880.92s | But if we need to scale the network 880.92s - 882.48s | up to thousands of relays, 882.48s - 885.28s | and they need to be in diverse locations around the world, 885.28s - 887.40s | I can't go meet


chunk_id: 214_chunk_18
chunk_text:
- 842.39s | And it's a way for us to give new tools 842.39s - 843.51s | to the relay community, 843.51s - 844.55s | because in the past, 844.55s - 847.67s | we noticed that they were organizing informally. 847.67s - 849.67s | And this is a tool to help them 849.67s - 854.67s | with managing expectation and with governance. 860.56s - 862.00s | Testing, testing, great. 862.00s - 864.40s | So where do these relays come from? 864.40s - 867.24s | And originally, it was people that I had met 867.24s - 869.16s | at hacker conferences like this. 869.16s - 871.00s | And I said, tour is really important. 871.00s - 872.72s | And you said, how can I help? 872.72s - 874.64s | And I convinced you to run a relay. 874.64s - 877.76s | And that worked great for the first 50 relays 877.76s - 879.00s | or 100 relays. 879.04s - 880.92s | But if we need to scale the network 880.92s - 882.48s | up to thousands of relays, 882.48s - 885.28s | and they need to be in diverse locations around the world, 885.28s - 887.40s | I can't go meet all of the people. 887.40s - 890.44s | In order to have an actually global safe network, 890.44s - 892.76s | we need to have them everywhere 892.76s - 895.12s | and run by people that we don't know as well. 895.12s - 896.28s | And the challenge there, 896.28s - 898.32s | especially in the past few years, 898.32s - 901.40s | the set of reasons why bad people 901.40s - 903.64s | might want to set up relays has grown. 903.64s - 907.04s | We've ended up with people in Russia setting up relays 907.04s - 911.40s | in order to do a man-in-the-middle attacks on exit traffic. 911.40s - 914.60s | We've ended up, we've seen people running relays, 914.60s - 916.00s | a lot of them for reasons 916.00s - 918.40s | that we still don't quite understand what's going on. 918.40s - 921.96s | So this is an ongoing and growing challenge. 921.96s - 923.76s | How do we make sure that the tour network 923.76s - 926.68s | has the relays that are doing it for the right reasons? 926.68s - 928.52s | And part of that, so how do we do that? 928.52s


chunk_id: 214_chunk_19
chunk_text:
all of the people. 887.40s - 890.44s | In order to have an actually global safe network, 890.44s - 892.76s | we need to have them everywhere 892.76s - 895.12s | and run by people that we don't know as well. 895.12s - 896.28s | And the challenge there, 896.28s - 898.32s | especially in the past few years, 898.32s - 901.40s | the set of reasons why bad people 901.40s - 903.64s | might want to set up relays has grown. 903.64s - 907.04s | We've ended up with people in Russia setting up relays 907.04s - 911.40s | in order to do a man-in-the-middle attacks on exit traffic. 911.40s - 914.60s | We've ended up, we've seen people running relays, 914.60s - 916.00s | a lot of them for reasons 916.00s - 918.40s | that we still don't quite understand what's going on. 918.40s - 921.96s | So this is an ongoing and growing challenge. 921.96s - 923.76s | How do we make sure that the tour network 923.76s - 926.68s | has the relays that are doing it for the right reasons? 926.68s - 928.52s | And part of that, so how do we do that? 928.52s - 930.32s | How do we actually accomplish that? 930.32s - 932.64s | The first answer is we need to get rid 932.64s - 936.08s | of application-level vulnerabilities. 936.08s - 939.04s | For example, the reason why those Russian people 939.04s - 941.48s | could run man-in-the-middle attacks 941.48s - 946.48s | is because some tour users were doing HTTP, not HTTPS. 946.64s - 949.08s | So we managed to fix the browser 949.08s - 952.32s | to do HTTPS by default always, 952.32s - 956.96s | and that is a structural way to resolve that sort of attack. 956.96s - 959.64s | Another angle to look at this, 959.64s - 962.88s | one of the few tools that we have in this community, 962.88s - 965.28s | in this social group of relays, 965.28s - 966.96s | is the social graph. 966.96s - 968.44s | It's who knows who. 968.44s - 971.76s | So if we, as a community of relay operators, 971.76s - 973.72s | get together and know each other, 973.72s - 977.58s | then we can be stronger against outsider or FBI people 977.58s - 979.52s | or whatever trying to come in and join.


chunk_id: 214_chunk_20
chunk_text:
- 930.32s | How do we actually accomplish that? 930.32s - 932.64s | The first answer is we need to get rid 932.64s - 936.08s | of application-level vulnerabilities. 936.08s - 939.04s | For example, the reason why those Russian people 939.04s - 941.48s | could run man-in-the-middle attacks 941.48s - 946.48s | is because some tour users were doing HTTP, not HTTPS. 946.64s - 949.08s | So we managed to fix the browser 949.08s - 952.32s | to do HTTPS by default always, 952.32s - 956.96s | and that is a structural way to resolve that sort of attack. 956.96s - 959.64s | Another angle to look at this, 959.64s - 962.88s | one of the few tools that we have in this community, 962.88s - 965.28s | in this social group of relays, 965.28s - 966.96s | is the social graph. 966.96s - 968.44s | It's who knows who. 968.44s - 971.76s | So if we, as a community of relay operators, 971.76s - 973.72s | get together and know each other, 973.72s - 977.58s | then we can be stronger against outsider or FBI people 977.58s - 979.52s | or whatever trying to come in and join. 980.64s - 984.12s | And speaking of that, one of the really successful models 984.12s - 986.32s | over the past 10 years or so 986.32s - 988.76s | is relay operator associations. 988.76s - 991.16s | These are non-profits around the world who, 991.16s - 994.88s | these are individuals who banded together in their country. 994.88s - 998.92s | One of them exists in France, in Switzerland, in Germany. 998.92s - 1000.16s | There's one in Sweden. 1000.16s - 1001.36s | There was one in Luxembourg. 1001.36s - 1002.40s | There was one in Iceland. 1002.40s - 1003.64s | There are two in Canada. 1003.64s - 1008.28s | It's a global phenomenon where non-profit hacker spaces 1008.28s - 1010.72s | get together to run tour exit relays. 1010.72s - 1012.52s | And one of the great things about this 1012.52s - 1014.14s | is it's sustainable. 1014.14s - 1016.00s | You can run really big relays. 1016.00s - 1018.92s | Another one is they can establish relationships 1018.92s - 1022.32s | with the local cops so they know how tour works. 1022.32s - 1023.80s | And also, we know who they are. 1023.80s - 1025.92s | We meet


chunk_id: 214_chunk_21
chunk_text:
980.64s - 984.12s | And speaking of that, one of the really successful models 984.12s - 986.32s | over the past 10 years or so 986.32s - 988.76s | is relay operator associations. 988.76s - 991.16s | These are non-profits around the world who, 991.16s - 994.88s | these are individuals who banded together in their country. 994.88s - 998.92s | One of them exists in France, in Switzerland, in Germany. 998.92s - 1000.16s | There's one in Sweden. 1000.16s - 1001.36s | There was one in Luxembourg. 1001.36s - 1002.40s | There was one in Iceland. 1002.40s - 1003.64s | There are two in Canada. 1003.64s - 1008.28s | It's a global phenomenon where non-profit hacker spaces 1008.28s - 1010.72s | get together to run tour exit relays. 1010.72s - 1012.52s | And one of the great things about this 1012.52s - 1014.14s | is it's sustainable. 1014.14s - 1016.00s | You can run really big relays. 1016.00s - 1018.92s | Another one is they can establish relationships 1018.92s - 1022.32s | with the local cops so they know how tour works. 1022.32s - 1023.80s | And also, we know who they are. 1023.80s - 1025.92s | We meet them at conferences like DEF CON 1025.92s - 1027.32s | and grow that community. 1027.32s - 1029.44s | So having them be part of our community 1029.44s - 1032.48s | is one of the ways we can handle that asymmetry 1032.48s - 1035.40s | where we need to have a lot of relays around the world, 1035.40s - 1037.76s | but also we need to know who they are 1037.76s - 1040.08s | and have them work together. 1040.08s - 1044.20s | And speaking of that, EFF a year ago launched a campaign 1044.20s - 1047.12s | to get universities in particular running relays. 1047.12s - 1049.52s | And part of the reason for that is universities, 1049.52s - 1051.96s | that's where freedom of speech, 1051.96s - 1054.20s | freedom of thought is supposed to be. 1054.20s - 1058.32s | But it's not just the freedom side. 1058.32s - 1060.36s | Universities are about education. 1060.36s - 1062.44s | You can run a relay at your university 1062.44s - 1066.40s | and learn how to put a computer on the actual internet, 1066.40s - 1069.76s | which is a thing that we all used to do. 1069.76s - 1072.44s


chunk_id: 214_chunk_22
chunk_text:
them at conferences like DEF CON 1025.92s - 1027.32s | and grow that community. 1027.32s - 1029.44s | So having them be part of our community 1029.44s - 1032.48s | is one of the ways we can handle that asymmetry 1032.48s - 1035.40s | where we need to have a lot of relays around the world, 1035.40s - 1037.76s | but also we need to know who they are 1037.76s - 1040.08s | and have them work together. 1040.08s - 1044.20s | And speaking of that, EFF a year ago launched a campaign 1044.20s - 1047.12s | to get universities in particular running relays. 1047.12s - 1049.52s | And part of the reason for that is universities, 1049.52s - 1051.96s | that's where freedom of speech, 1051.96s - 1054.20s | freedom of thought is supposed to be. 1054.20s - 1058.32s | But it's not just the freedom side. 1058.32s - 1060.36s | Universities are about education. 1060.36s - 1062.44s | You can run a relay at your university 1062.44s - 1066.40s | and learn how to put a computer on the actual internet, 1066.40s - 1069.76s | which is a thing that we all used to do. 1069.76s - 1072.44s | And now the cloud exists, so not as many people do that. 1072.44s - 1074.12s | So there's an education side. 1074.12s - 1075.24s | There's a community side. 1075.24s - 1076.52s | There's a research side. 1076.52s - 1078.04s | If you want to help do that, 1078.04s - 1080.40s | they have these amazing coins 1080.40s - 1082.12s | that they created a little while ago 1082.12s - 1084.08s | that I'll show you afterwards, 1084.08s - 1086.42s | that if you run a relay at your university, 1086.42s - 1088.72s | they will send you one of these amazing coins. 1089.72s - 1093.46s | And relay incentives in general is a broad topic 1093.46s - 1095.32s | that we really need to work harder on. 1095.32s - 1100.32s | So gamification, like Reddit style, Stack Exchange style. 1100.96s - 1104.24s | I got a gold badge for my five-year anniversary. 1104.24s - 1106.84s | I got a silver badge because I was an exit relay. 1106.84s - 1110.24s | I got a silver badge because I'm in an ISP 1111.08s - 1112.32s | that there aren't very many other relays in. 1112.32s - 1116.60s | So


chunk_id: 214_chunk_23
chunk_text:
| And now the cloud exists, so not as many people do that. 1072.44s - 1074.12s | So there's an education side. 1074.12s - 1075.24s | There's a community side. 1075.24s - 1076.52s | There's a research side. 1076.52s - 1078.04s | If you want to help do that, 1078.04s - 1080.40s | they have these amazing coins 1080.40s - 1082.12s | that they created a little while ago 1082.12s - 1084.08s | that I'll show you afterwards, 1084.08s - 1086.42s | that if you run a relay at your university, 1086.42s - 1088.72s | they will send you one of these amazing coins. 1089.72s - 1093.46s | And relay incentives in general is a broad topic 1093.46s - 1095.32s | that we really need to work harder on. 1095.32s - 1100.32s | So gamification, like Reddit style, Stack Exchange style. 1100.96s - 1104.24s | I got a gold badge for my five-year anniversary. 1104.24s - 1106.84s | I got a silver badge because I was an exit relay. 1106.84s - 1110.24s | I got a silver badge because I'm in an ISP 1111.08s - 1112.32s | that there aren't very many other relays in. 1112.32s - 1116.60s | So coming up with ways to highlight what we need most 1116.60s - 1119.00s | and make people feel happy and part of a community 1119.00s - 1120.48s | and check out my gold badge. 1120.48s - 1122.08s | I can put this on Twitter. 1122.08s - 1124.76s | Leaderboards, I'm running the 17th fastest 1124.76s - 1126.56s | tour relay in the world right now. 1126.56s - 1127.80s | So that's one piece. 1127.80s - 1130.56s | Another piece is just giving people more attention. 1130.56s - 1134.60s | Meetups, having a time to answer their questions 1134.60s - 1137.00s | and walk people through how to run relays. 1137.00s - 1139.28s | And then there are the structural incentives. 1139.64s - 1143.28s | Imagine we had a tour design where if you run a relay, 1143.28s - 1145.62s | you get better performance on tour. 1145.62s - 1149.02s | So there are a bunch of exciting security problems 1149.02s - 1150.84s | that come with a lot of these designs, 1150.84s - 1153.14s | but maybe there's a version of this design 1153.14s - 1156.08s | where we can get rid of the security flaws 1156.08s - 1158.02s |


chunk_id: 214_chunk_24
chunk_text:
coming up with ways to highlight what we need most 1116.60s - 1119.00s | and make people feel happy and part of a community 1119.00s - 1120.48s | and check out my gold badge. 1120.48s - 1122.08s | I can put this on Twitter. 1122.08s - 1124.76s | Leaderboards, I'm running the 17th fastest 1124.76s - 1126.56s | tour relay in the world right now. 1126.56s - 1127.80s | So that's one piece. 1127.80s - 1130.56s | Another piece is just giving people more attention. 1130.56s - 1134.60s | Meetups, having a time to answer their questions 1134.60s - 1137.00s | and walk people through how to run relays. 1137.00s - 1139.28s | And then there are the structural incentives. 1139.64s - 1143.28s | Imagine we had a tour design where if you run a relay, 1143.28s - 1145.62s | you get better performance on tour. 1145.62s - 1149.02s | So there are a bunch of exciting security problems 1149.02s - 1150.84s | that come with a lot of these designs, 1150.84s - 1153.14s | but maybe there's a version of this design 1153.14s - 1156.08s | where we can get rid of the security flaws 1156.08s - 1158.02s | while still keeping the incentives. 1158.02s - 1161.68s | And then the last topic is a lot of cryptocurrency people 1161.68s - 1164.32s | these days keep saying, you know what you need to do? 1164.32s - 1167.04s | You need an incentive coin thing 1167.04s - 1169.00s | so that people will want to run relays 1169.72s - 1170.56s | because they'll make money. 1170.56s - 1173.08s | They'll make Bitcoins, dollars, and so on. 1173.08s - 1176.88s | And part of the challenge there is the cryptocurrency world, 1176.88s - 1179.80s | their fundamental building block is capitalism, 1179.80s - 1181.42s | is maximizing profit. 1181.42s - 1182.60s | And the tour world, 1182.60s - 1185.24s | our fundamental building block is altruism. 1185.24s - 1187.36s | It's I want to make the world a better place. 1187.36s - 1188.96s | I have extra resources. 1188.96s - 1190.32s | I can provide them. 1190.32s - 1195.32s | So these are very different worlds in terms of organic. 1196.32s - 1199.08s | Do you build an organic community of people 1199.08s - 1200.48s | who want to help each other? 1200.48s - 1203.24s | Or are you just trying


chunk_id: 214_chunk_25
chunk_text:
while still keeping the incentives. 1158.02s - 1161.68s | And then the last topic is a lot of cryptocurrency people 1161.68s - 1164.32s | these days keep saying, you know what you need to do? 1164.32s - 1167.04s | You need an incentive coin thing 1167.04s - 1169.00s | so that people will want to run relays 1169.72s - 1170.56s | because they'll make money. 1170.56s - 1173.08s | They'll make Bitcoins, dollars, and so on. 1173.08s - 1176.88s | And part of the challenge there is the cryptocurrency world, 1176.88s - 1179.80s | their fundamental building block is capitalism, 1179.80s - 1181.42s | is maximizing profit. 1181.42s - 1182.60s | And the tour world, 1182.60s - 1185.24s | our fundamental building block is altruism. 1185.24s - 1187.36s | It's I want to make the world a better place. 1187.36s - 1188.96s | I have extra resources. 1188.96s - 1190.32s | I can provide them. 1190.32s - 1195.32s | So these are very different worlds in terms of organic. 1196.32s - 1199.08s | Do you build an organic community of people 1199.08s - 1200.48s | who want to help each other? 1200.48s - 1203.24s | Or are you just trying to everybody for themselves 1203.24s - 1204.56s | maximize your profit? 1204.56s - 1207.12s | And in that case, if you want to do it for profit, 1207.12s - 1209.68s | maybe you'll all go to the cheapest hosting provider 1209.68s - 1212.06s | so you can spend as little money as possible. 1212.06s - 1214.00s | Or maybe you'll sell your data on the side 1214.00s - 1215.24s | to make a little bit more money 1215.24s - 1218.28s | because capitalism means maximizing profit. 1218.28s - 1223.28s | So these are fundamentally incompatible approaches 1223.68s - 1227.48s | to how to build a community of volunteers around the world 1227.48s - 1232.16s | who are trying to keep people safe. 1232.16s - 1236.00s | Okay, so I want to tell you a bit about network health 1236.00s - 1238.12s | and metrics and what we do. 1238.12s - 1240.00s | But first I want to tell you about a story 1240.00s - 1243.56s | because I want to put what we do in context. 1243.56s - 1246.44s | So between May and June in 2020, 1246.44s - 1250.48s | we removed a group of Torexit relays 1250.48s -


chunk_id: 214_chunk_26
chunk_text:
to everybody for themselves 1203.24s - 1204.56s | maximize your profit? 1204.56s - 1207.12s | And in that case, if you want to do it for profit, 1207.12s - 1209.68s | maybe you'll all go to the cheapest hosting provider 1209.68s - 1212.06s | so you can spend as little money as possible. 1212.06s - 1214.00s | Or maybe you'll sell your data on the side 1214.00s - 1215.24s | to make a little bit more money 1215.24s - 1218.28s | because capitalism means maximizing profit. 1218.28s - 1223.28s | So these are fundamentally incompatible approaches 1223.68s - 1227.48s | to how to build a community of volunteers around the world 1227.48s - 1232.16s | who are trying to keep people safe. 1232.16s - 1236.00s | Okay, so I want to tell you a bit about network health 1236.00s - 1238.12s | and metrics and what we do. 1238.12s - 1240.00s | But first I want to tell you about a story 1240.00s - 1243.56s | because I want to put what we do in context. 1243.56s - 1246.44s | So between May and June in 2020, 1246.44s - 1250.48s | we removed a group of Torexit relays 1250.48s - 1253.16s | that were doing an SSL strip attack. 1253.16s - 1254.84s | And we didn't spot this at first 1254.84s - 1258.32s | because they were leaving all the exit traffic alone, 1258.32s - 1260.68s | but they were intercepting just a little bit of the traffic 1260.68s - 1265.08s | that was going to some cryptocurrency exchange websites. 1265.08s - 1267.92s | So what this were doing was that 1269.18s - 1272.56s | if the user was visiting the HTTP website, 1272.56s - 1275.96s | not the ghttps one, they wouldn't redirect the user. 1275.96s - 1278.84s | And if they wouldn't notice that there was not a lock, 1278.84s - 1281.64s | for example, in the browser, in the tab, 1281.64s - 1285.44s | they would just continue to send their login information, 1285.44s - 1288.76s | for example, and this information would be intercepted. 1288.76s - 1292.28s | So we noticed this, we kicked the relays out. 1292.28s - 1294.20s | And then a few weeks later in June, 1294.20s - 1297.28s | another group of relays does the same attack. 1297.28s - 1300.88s | And we noticed again, and we kicked them out. 1300.88s -


chunk_id: 214_chunk_27
chunk_text:
1253.16s | that were doing an SSL strip attack. 1253.16s - 1254.84s | And we didn't spot this at first 1254.84s - 1258.32s | because they were leaving all the exit traffic alone, 1258.32s - 1260.68s | but they were intercepting just a little bit of the traffic 1260.68s - 1265.08s | that was going to some cryptocurrency exchange websites. 1265.08s - 1267.92s | So what this were doing was that 1269.18s - 1272.56s | if the user was visiting the HTTP website, 1272.56s - 1275.96s | not the ghttps one, they wouldn't redirect the user. 1275.96s - 1278.84s | And if they wouldn't notice that there was not a lock, 1278.84s - 1281.64s | for example, in the browser, in the tab, 1281.64s - 1285.44s | they would just continue to send their login information, 1285.44s - 1288.76s | for example, and this information would be intercepted. 1288.76s - 1292.28s | So we noticed this, we kicked the relays out. 1292.28s - 1294.20s | And then a few weeks later in June, 1294.20s - 1297.28s | another group of relays does the same attack. 1297.28s - 1300.88s | And we noticed again, and we kicked them out. 1300.88s - 1303.24s | And then we realized that network health 1303.24s - 1305.86s | is not just about kicking relays out, 1305.86s - 1306.96s | although that's important 1306.96s - 1310.04s | because you don't want malicious relay on the network. 1310.28s - 1312.36s | It's a little bit more. 1312.36s - 1314.80s | So we got together with the browser team 1314.80s - 1319.12s | and because of projects like HTTPS Everywhere 1319.12s - 1321.32s | and a newer version of Firefox 1321.32s - 1325.80s | that allowed the browser to default to HTTPS 1325.80s - 1330.52s | and then fall back to HTTP just when needed, 1330.52s - 1334.20s | we were able to enable HTTPS by default into our browser 1334.20s - 1337.08s | and protect our user, not just the network level, 1337.08s - 1339.60s | but also the application level. 1339.60s - 1343.48s | And so out of incident like this, 1343.48s - 1347.52s | that we understood the value of having a network health team 1347.52s - 1350.56s | which is basically joining together a group of people 1350.56s - 1352.94s | that keep track of the network. 1352.94s - 1357.08s | And part of


chunk_id: 214_chunk_28
chunk_text:
1303.24s | And then we realized that network health 1303.24s - 1305.86s | is not just about kicking relays out, 1305.86s - 1306.96s | although that's important 1306.96s - 1310.04s | because you don't want malicious relay on the network. 1310.28s - 1312.36s | It's a little bit more. 1312.36s - 1314.80s | So we got together with the browser team 1314.80s - 1319.12s | and because of projects like HTTPS Everywhere 1319.12s - 1321.32s | and a newer version of Firefox 1321.32s - 1325.80s | that allowed the browser to default to HTTPS 1325.80s - 1330.52s | and then fall back to HTTP just when needed, 1330.52s - 1334.20s | we were able to enable HTTPS by default into our browser 1334.20s - 1337.08s | and protect our user, not just the network level, 1337.08s - 1339.60s | but also the application level. 1339.60s - 1343.48s | And so out of incident like this, 1343.48s - 1347.52s | that we understood the value of having a network health team 1347.52s - 1350.56s | which is basically joining together a group of people 1350.56s - 1352.94s | that keep track of the network. 1352.94s - 1357.08s | And part of what we do is kicking relays out of the network. 1357.08s - 1362.00s | So, but also part of what we do is community engagement, 1362.00s - 1363.68s | is working with the relay community, 1363.68s - 1367.84s | working with activists sometimes in the ground. 1367.84s - 1371.20s | And the foundation of what we do also 1371.20s - 1373.68s | is collecting the metrics about the Tor network 1373.68s - 1377.04s | that gives us an idea of how the network is performing, 1377.04s - 1378.36s | how the network is doing 1378.36s - 1381.50s | and to understand if something is not right. 1382.60s - 1387.24s | And so when there is a geopolitical event, for example, 1387.24s - 1388.70s | we are able to spot it 1388.70s - 1392.56s | because we are able to notice some patterns on the network. 1392.56s - 1397.32s | And in this case, this is a picture of, 1397.32s - 1400.12s | it's a graph of directly connecting user from Russia 1400.12s - 1404.06s | in between 2021 and 2022. 1404.06s - 1406.68s | And we start noticing that there was censorship 1406.68s - 1411.04s | being applied in Russia


chunk_id: 214_chunk_29
chunk_text:
what we do is kicking relays out of the network. 1357.08s - 1362.00s | So, but also part of what we do is community engagement, 1362.00s - 1363.68s | is working with the relay community, 1363.68s - 1367.84s | working with activists sometimes in the ground. 1367.84s - 1371.20s | And the foundation of what we do also 1371.20s - 1373.68s | is collecting the metrics about the Tor network 1373.68s - 1377.04s | that gives us an idea of how the network is performing, 1377.04s - 1378.36s | how the network is doing 1378.36s - 1381.50s | and to understand if something is not right. 1382.60s - 1387.24s | And so when there is a geopolitical event, for example, 1387.24s - 1388.70s | we are able to spot it 1388.70s - 1392.56s | because we are able to notice some patterns on the network. 1392.56s - 1397.32s | And in this case, this is a picture of, 1397.32s - 1400.12s | it's a graph of directly connecting user from Russia 1400.12s - 1404.06s | in between 2021 and 2022. 1404.06s - 1406.68s | And we start noticing that there was censorship 1406.68s - 1411.04s | being applied in Russia between November and December, 1411.04s - 1413.64s | which a few months before the Ukraine invasion. 1414.56s - 1417.72s | And we saw directly connect user decline. 1417.72s - 1421.40s | And as we saw this happening, 1421.40s - 1425.48s | we also were able to work with the anti-censorship team, 1425.48s - 1426.84s | with activists on the ground 1427.36s - 1430.16s | and we were able to tell the user to use bridges. 1430.16s - 1433.72s | And so we saw a spike in bridge users in Russia. 1433.72s - 1435.84s | And that meant that we are able to help these people 1435.84s - 1437.84s | reach the news and know what is happening 1437.84s - 1439.08s | outside the country. 1440.74s - 1443.22s | Yeah, so let me jump in and explain what a bridge is 1443.22s - 1444.76s | for those who don't know. 1444.76s - 1446.76s | We've been talking about relays so far. 1446.76s - 1449.82s | These are public Tor relays that are in the list. 1449.82s - 1451.92s | And that means that the censors can get that list 1451.92s - 1453.12s | and block all of them. 1453.12s - 1456.68s


chunk_id: 214_chunk_30
chunk_text:
between November and December, 1411.04s - 1413.64s | which a few months before the Ukraine invasion. 1414.56s - 1417.72s | And we saw directly connect user decline. 1417.72s - 1421.40s | And as we saw this happening, 1421.40s - 1425.48s | we also were able to work with the anti-censorship team, 1425.48s - 1426.84s | with activists on the ground 1427.36s - 1430.16s | and we were able to tell the user to use bridges. 1430.16s - 1433.72s | And so we saw a spike in bridge users in Russia. 1433.72s - 1435.84s | And that meant that we are able to help these people 1435.84s - 1437.84s | reach the news and know what is happening 1437.84s - 1439.08s | outside the country. 1440.74s - 1443.22s | Yeah, so let me jump in and explain what a bridge is 1443.22s - 1444.76s | for those who don't know. 1444.76s - 1446.76s | We've been talking about relays so far. 1446.76s - 1449.82s | These are public Tor relays that are in the list. 1449.82s - 1451.92s | And that means that the censors can get that list 1451.92s - 1453.12s | and block all of them. 1453.12s - 1456.68s | So bridges are relays that aren't in the list. 1457.50s - 1459.52s | And so the goal is to transform the traffic in some way 1459.52s - 1460.82s | so that it's doing a protocol 1460.82s - 1462.40s | the censors don't want to block 1462.40s - 1464.32s | and it's talking to an IP address 1464.32s - 1466.14s | that the censors don't think is Tor. 1466.14s - 1469.08s | So that's the arms race that we're in there. 1469.08s - 1471.12s | And there are a bunch of different approaches 1471.12s - 1472.48s | we have for bridges. 1472.48s - 1476.18s | One of which is? 1476.18s - 1481.18s | Yeah, and similar situation in Iran in 2022, 1481.40s - 1483.42s | the Massimini protests. 1483.42s - 1488.42s | So people, we started noticing that there was aggressive 1488.84s - 1490.64s | censorship happening in the country. 1490.64s - 1492.56s | People were in the streets protesting. 1492.56s - 1494.80s | They couldn't access the internet. 1494.80s - 1496.24s | They couldn't access the news. 1496.24s - 1500.20s | They couldn't use communication messaging systems. 1500.20s - 1503.90s | So by working again with people on the


chunk_id: 214_chunk_31
chunk_text:
| So bridges are relays that aren't in the list. 1457.50s - 1459.52s | And so the goal is to transform the traffic in some way 1459.52s - 1460.82s | so that it's doing a protocol 1460.82s - 1462.40s | the censors don't want to block 1462.40s - 1464.32s | and it's talking to an IP address 1464.32s - 1466.14s | that the censors don't think is Tor. 1466.14s - 1469.08s | So that's the arms race that we're in there. 1469.08s - 1471.12s | And there are a bunch of different approaches 1471.12s - 1472.48s | we have for bridges. 1472.48s - 1476.18s | One of which is? 1476.18s - 1481.18s | Yeah, and similar situation in Iran in 2022, 1481.40s - 1483.42s | the Massimini protests. 1483.42s - 1488.42s | So people, we started noticing that there was aggressive 1488.84s - 1490.64s | censorship happening in the country. 1490.64s - 1492.56s | People were in the streets protesting. 1492.56s - 1494.80s | They couldn't access the internet. 1494.80s - 1496.24s | They couldn't access the news. 1496.24s - 1500.20s | They couldn't use communication messaging systems. 1500.20s - 1503.90s | So by working again with people on the ground, 1503.90s - 1505.90s | the activists, the local people 1505.90s - 1507.28s | and the anti-censorship team, 1507.28s - 1511.70s | we were able to teach these users how to use Snowflake. 1511.70s - 1516.70s | And we saw how the users from Iran spiked 1516.90s - 1518.66s | during the protest. 1519.42s - 1520.26s | And for these people, 1520.26s - 1522.14s | reaching Tor was a lifeline in a way 1522.14s - 1524.14s | because they were able to communicate 1524.14s - 1527.34s | between one another in ways that it was not possible 1527.34s - 1529.70s | at the moment with any other tool. 1529.70s - 1531.62s | And if you're wondering what Snowflake is, 1531.62s - 1533.50s | so in the Tor ecosystem, 1533.50s - 1536.70s | Snowflake is what we call a pluggable transport. 1536.70s - 1539.74s | And you can think of it like as a tool 1539.74s - 1541.84s | that Tor used to disguise the traffic. 1541.84s - 1543.84s | And in the case of Snowflake, 1543.84s - 1546.94s | it used temporary run volunteer proxies. 1546.94s - 1550.38s | So it's by using the proxies, 1550.38s - 1553.82s | more


chunk_id: 214_chunk_32
chunk_text:
ground, 1503.90s - 1505.90s | the activists, the local people 1505.90s - 1507.28s | and the anti-censorship team, 1507.28s - 1511.70s | we were able to teach these users how to use Snowflake. 1511.70s - 1516.70s | And we saw how the users from Iran spiked 1516.90s - 1518.66s | during the protest. 1519.42s - 1520.26s | And for these people, 1520.26s - 1522.14s | reaching Tor was a lifeline in a way 1522.14s - 1524.14s | because they were able to communicate 1524.14s - 1527.34s | between one another in ways that it was not possible 1527.34s - 1529.70s | at the moment with any other tool. 1529.70s - 1531.62s | And if you're wondering what Snowflake is, 1531.62s - 1533.50s | so in the Tor ecosystem, 1533.50s - 1536.70s | Snowflake is what we call a pluggable transport. 1536.70s - 1539.74s | And you can think of it like as a tool 1539.74s - 1541.84s | that Tor used to disguise the traffic. 1541.84s - 1543.84s | And in the case of Snowflake, 1543.84s - 1546.94s | it used temporary run volunteer proxies. 1546.94s - 1550.38s | So it's by using the proxies, 1550.38s - 1553.82s | more difficult for the sensors to actually block them. 1553.82s - 1557.90s | And it helps people in very difficult situation 1557.90s - 1559.20s | to reach the internet. 1560.94s - 1562.74s | Yeah, so two things to point out 1562.74s - 1564.10s | for this graph in particular. 1564.10s - 1566.24s | One of them is that y-axis, 1566.24s - 1568.10s | we're talking a hundred thousand 1568.10s - 1570.06s | or maybe hundreds of thousands 1570.06s - 1573.10s | of people using Snowflake in Iran during that event. 1573.10s - 1576.06s | So as we're showing more recent graphs, 1576.06s - 1578.22s | we end up with larger and larger numbers. 1578.22s - 1580.86s | We're talking about many tens of thousands 1580.86s - 1582.42s | or hundreds of thousands of people. 1582.42s - 1585.84s | And the other answer is snowflake.torproject.org. 1585.84s - 1588.06s | You too can volunteer as a Snowflake. 1588.06s - 1590.26s | You can either install it as a browser extension 1590.26s - 1591.50s | or there's a headless version 1591.50s - 1593.74s | for your Linux system based on Go. 1593.74s - 1596.92s | Please help. 1596.92s - 1599.32s | And so more


chunk_id: 214_chunk_33
chunk_text:
difficult for the sensors to actually block them. 1553.82s - 1557.90s | And it helps people in very difficult situation 1557.90s - 1559.20s | to reach the internet. 1560.94s - 1562.74s | Yeah, so two things to point out 1562.74s - 1564.10s | for this graph in particular. 1564.10s - 1566.24s | One of them is that y-axis, 1566.24s - 1568.10s | we're talking a hundred thousand 1568.10s - 1570.06s | or maybe hundreds of thousands 1570.06s - 1573.10s | of people using Snowflake in Iran during that event. 1573.10s - 1576.06s | So as we're showing more recent graphs, 1576.06s - 1578.22s | we end up with larger and larger numbers. 1578.22s - 1580.86s | We're talking about many tens of thousands 1580.86s - 1582.42s | or hundreds of thousands of people. 1582.42s - 1585.84s | And the other answer is snowflake.torproject.org. 1585.84s - 1588.06s | You too can volunteer as a Snowflake. 1588.06s - 1590.26s | You can either install it as a browser extension 1590.26s - 1591.50s | or there's a headless version 1591.50s - 1593.74s | for your Linux system based on Go. 1593.74s - 1596.92s | Please help. 1596.92s - 1599.32s | And so more recent case, 1599.32s - 1603.68s | we saw just a few months ago between May and June, 1603.68s - 1607.64s | we started to see user spikes in Myanmar. 1607.64s - 1610.52s | And then we got news that the military junta 1610.52s - 1613.48s | and started to apply internet shutdown. 1613.48s - 1616.80s | So again, what we did was get together with the browser, 1616.80s - 1618.38s | the anti-censorship team, 1618.38s - 1622.04s | and help users connecting to Tor 1622.04s - 1625.50s | by distributing bridges directly for them in the country. 1625.50s - 1629.04s | And we saw how from not having 1629.04s - 1631.14s | almost zero bridge users in the country, 1631.14s - 1638.96s | we got a spike in a few days to just half a thousand. 1639.16s - 1641.24s | So one of the fascinating things 1641.24s - 1642.90s | about the graph like this, 1642.90s - 1645.92s | periodically we see spikes in our user graphs. 1645.92s - 1648.16s | And here we're like, wow, a whole lot of people 1648.16s - 1651.68s | from Myanmar just started using Tor over bridges. 1651.68s - 1652.60s | This is great.


chunk_id: 214_chunk_34
chunk_text:
recent case, 1599.32s - 1603.68s | we saw just a few months ago between May and June, 1603.68s - 1607.64s | we started to see user spikes in Myanmar. 1607.64s - 1610.52s | And then we got news that the military junta 1610.52s - 1613.48s | and started to apply internet shutdown. 1613.48s - 1616.80s | So again, what we did was get together with the browser, 1616.80s - 1618.38s | the anti-censorship team, 1618.38s - 1622.04s | and help users connecting to Tor 1622.04s - 1625.50s | by distributing bridges directly for them in the country. 1625.50s - 1629.04s | And we saw how from not having 1629.04s - 1631.14s | almost zero bridge users in the country, 1631.14s - 1638.96s | we got a spike in a few days to just half a thousand. 1639.16s - 1641.24s | So one of the fascinating things 1641.24s - 1642.90s | about the graph like this, 1642.90s - 1645.92s | periodically we see spikes in our user graphs. 1645.92s - 1648.16s | And here we're like, wow, a whole lot of people 1648.16s - 1651.68s | from Myanmar just started using Tor over bridges. 1651.68s - 1652.60s | This is great. 1652.60s - 1654.08s | But sometimes we see the spikes 1654.08s - 1656.44s | and we don't know if it actually happened that way. 1656.44s - 1658.44s | We don't know what happened in the country. 1658.44s - 1660.08s | So every time we see a spike like this, 1660.08s - 1662.04s | we're wondering, is there a bug? 1662.04s - 1664.72s | Is there like a GOIP database mess up 1664.72s - 1666.90s | where actually these are people in Germany 1666.90s - 1670.34s | who got misclassified as being in Myanmar for that week? 1670.34s - 1671.74s | So part of the challenge we have 1671.74s - 1674.06s | is understanding what actually happened. 1674.06s - 1676.58s | And one of the principles that we have for that situation 1676.58s - 1679.46s | is trying to learn the same data points 1679.46s - 1681.14s | from a bunch of different dimensions, 1681.14s - 1682.86s | from a bunch of different angles, 1682.86s - 1685.36s | so that we can confirm that what we're seeing 1685.36s - 1686.86s | actually really did happen. 1686.86s - 1688.42s | And part of that is angles, 1688.42s - 1691.10s | including talking


chunk_id: 214_chunk_35
chunk_text:
1652.60s - 1654.08s | But sometimes we see the spikes 1654.08s - 1656.44s | and we don't know if it actually happened that way. 1656.44s - 1658.44s | We don't know what happened in the country. 1658.44s - 1660.08s | So every time we see a spike like this, 1660.08s - 1662.04s | we're wondering, is there a bug? 1662.04s - 1664.72s | Is there like a GOIP database mess up 1664.72s - 1666.90s | where actually these are people in Germany 1666.90s - 1670.34s | who got misclassified as being in Myanmar for that week? 1670.34s - 1671.74s | So part of the challenge we have 1671.74s - 1674.06s | is understanding what actually happened. 1674.06s - 1676.58s | And one of the principles that we have for that situation 1676.58s - 1679.46s | is trying to learn the same data points 1679.46s - 1681.14s | from a bunch of different dimensions, 1681.14s - 1682.86s | from a bunch of different angles, 1682.86s - 1685.36s | so that we can confirm that what we're seeing 1685.36s - 1686.86s | actually really did happen. 1686.86s - 1688.42s | And part of that is angles, 1688.42s - 1691.10s | including talking to people on the ground and saying, 1691.10s - 1692.94s | what just happened in your country? 1692.94s - 1695.34s | Why is it that we're seeing these graphs, 1695.34s - 1699.78s | these changes in our graphs? 1699.78s - 1703.70s | Yeah, so we've been talking about user data 1703.70s - 1710.06s | and you might have wonder... 1710.06s - 1710.90s | That's loud. 1711.94s - 1712.78s | Okay. 1714.66s - 1715.50s | All right, go on. 1715.50s - 1716.32s | Okay. 1716.32s - 1717.70s | I think they can hear us better than we can. 1717.70s - 1718.54s | Okay. 1718.54s - 1722.54s | So you might wonder, Tor is the anonymity network. 1722.54s - 1724.38s | How come you have user data? 1724.38s - 1727.18s | And so the simple answer to that 1727.18s - 1729.82s | is that we live by a principle 1729.82s - 1731.42s | that is we do not collect data 1731.94s - 1733.18s | that cannot be public. 1733.18s - 1736.90s | So all our data that we have in metrics 1736.90s - 1739.14s | is actually public data that we publish. 1740.58s - 1743.34s | So we do not collect in-client


chunk_id: 214_chunk_36
chunk_text:
to people on the ground and saying, 1691.10s - 1692.94s | what just happened in your country? 1692.94s - 1695.34s | Why is it that we're seeing these graphs, 1695.34s - 1699.78s | these changes in our graphs? 1699.78s - 1703.70s | Yeah, so we've been talking about user data 1703.70s - 1710.06s | and you might have wonder... 1710.06s - 1710.90s | That's loud. 1711.94s - 1712.78s | Okay. 1714.66s - 1715.50s | All right, go on. 1715.50s - 1716.32s | Okay. 1716.32s - 1717.70s | I think they can hear us better than we can. 1717.70s - 1718.54s | Okay. 1718.54s - 1722.54s | So you might wonder, Tor is the anonymity network. 1722.54s - 1724.38s | How come you have user data? 1724.38s - 1727.18s | And so the simple answer to that 1727.18s - 1729.82s | is that we live by a principle 1729.82s - 1731.42s | that is we do not collect data 1731.94s - 1733.18s | that cannot be public. 1733.18s - 1736.90s | So all our data that we have in metrics 1736.90s - 1739.14s | is actually public data that we publish. 1740.58s - 1743.34s | So we do not collect in-client data. 1743.34s - 1745.38s | Our user data that we've been talking about 1745.38s - 1746.58s | are actual estimations. 1746.58s - 1748.90s | And I'm going to say how we do this. 1748.90s - 1751.34s | But other products like VPN products, 1751.34s - 1755.06s | they actually count the users and we don't. 1755.06s - 1758.62s | So what we have is aggregated data from relays. 1758.62s - 1762.50s | So the Tor browser or Tor clients in general, 1762.50s - 1763.74s | when they connect to the network, 1763.74s - 1766.38s | they request a list of relays information 1766.38s - 1768.50s | and they need this to build the circuits. 1768.50s - 1772.14s | So we estimate that if... 1773.30s - 1775.82s | First of all, we count all these requests globally, 1775.82s - 1777.90s | how they are collected from relays. 1777.90s - 1780.06s | And then we estimate that if a Tor client 1780.06s - 1784.70s | was connected 24 hours, it will make 10 requests. 1784.70s - 1788.10s | So we count all the requests that we get from the relays 1789.10s - 1791.74s | when they upload their statistics. 1791.74s - 1796.30s


chunk_id: 214_chunk_37
chunk_text:
data. 1743.34s - 1745.38s | Our user data that we've been talking about 1745.38s - 1746.58s | are actual estimations. 1746.58s - 1748.90s | And I'm going to say how we do this. 1748.90s - 1751.34s | But other products like VPN products, 1751.34s - 1755.06s | they actually count the users and we don't. 1755.06s - 1758.62s | So what we have is aggregated data from relays. 1758.62s - 1762.50s | So the Tor browser or Tor clients in general, 1762.50s - 1763.74s | when they connect to the network, 1763.74s - 1766.38s | they request a list of relays information 1766.38s - 1768.50s | and they need this to build the circuits. 1768.50s - 1772.14s | So we estimate that if... 1773.30s - 1775.82s | First of all, we count all these requests globally, 1775.82s - 1777.90s | how they are collected from relays. 1777.90s - 1780.06s | And then we estimate that if a Tor client 1780.06s - 1784.70s | was connected 24 hours, it will make 10 requests. 1784.70s - 1788.10s | So we count all the requests that we get from the relays 1789.10s - 1791.74s | when they upload their statistics. 1791.74s - 1796.30s | And we divide them by 10 and we have the user estimation. 1796.30s - 1799.54s | So in reality, we might be underestimating our user count 1799.54s - 1801.66s | by an order of magnitude. 1801.66s - 1803.38s | But that's fine for us. 1803.38s - 1805.02s | It's the trade-off that we want to make 1805.02s - 1808.42s | in order not to put our user at risk. 1808.42s - 1812.98s | So we do have, though, historical data. 1812.98s - 1816.86s | And we actually serve this data 1816.86s - 1818.74s | on collector.torproject.org. 1818.74s - 1821.18s | You find all the documents that have ever, I think, 1821.18s - 1823.30s | been created on the network. 1823.30s - 1827.62s | And out of this data, we have network-wide statistics. 1827.62s - 1830.82s | And we have also the status of individual nodes 1830.82s - 1834.74s | that will give us an idea of how the network is doing. 1834.74s - 1836.26s | And then something that we do 1836.26s - 1839.74s | is making the data accessible to the public, 1839.74s - 1842.42s | but also presenting the documentation 1842.42s - 1845.34s | of how we


chunk_id: 214_chunk_38
chunk_text:
| And we divide them by 10 and we have the user estimation. 1796.30s - 1799.54s | So in reality, we might be underestimating our user count 1799.54s - 1801.66s | by an order of magnitude. 1801.66s - 1803.38s | But that's fine for us. 1803.38s - 1805.02s | It's the trade-off that we want to make 1805.02s - 1808.42s | in order not to put our user at risk. 1808.42s - 1812.98s | So we do have, though, historical data. 1812.98s - 1816.86s | And we actually serve this data 1816.86s - 1818.74s | on collector.torproject.org. 1818.74s - 1821.18s | You find all the documents that have ever, I think, 1821.18s - 1823.30s | been created on the network. 1823.30s - 1827.62s | And out of this data, we have network-wide statistics. 1827.62s - 1830.82s | And we have also the status of individual nodes 1830.82s - 1834.74s | that will give us an idea of how the network is doing. 1834.74s - 1836.26s | And then something that we do 1836.26s - 1839.74s | is making the data accessible to the public, 1839.74s - 1842.42s | but also presenting the documentation 1842.42s - 1845.34s | of how we estimate the metrics that we have. 1845.34s - 1848.86s | And we believe in algorithm transparency 1848.86s - 1850.90s | so that if you download these tarballs 1850.90s - 1852.54s | and you process this data, 1852.54s - 1854.90s | you can come up with the same results as we do. 1854.90s - 1856.66s | And if you don't, you explain to us 1856.66s - 1858.14s | why you don't have the same results 1858.14s - 1862.22s | and maybe we have to change something on our side. 1862.22s - 1865.70s | So this is an overview of how this process works. 1865.70s - 1868.74s | So the relays have this little piece of information 1868.74s - 1871.02s | that they upload to directory authorities, 1871.02s - 1874.34s | which are special nodes in the network. 1874.34s - 1876.42s | And there is a service that's called collector 1877.38s - 1882.30s | which downloads these documents and archives them. 1882.30s - 1886.30s | And then we have a service that takes all these documents, 1886.30s - 1888.02s | parses them, puts them in a database 1888.02s - 1891.62s | and extracts time series and network statistics. 1891.62s - 1894.66s |


chunk_id: 214_chunk_39
chunk_text:
estimate the metrics that we have. 1845.34s - 1848.86s | And we believe in algorithm transparency 1848.86s - 1850.90s | so that if you download these tarballs 1850.90s - 1852.54s | and you process this data, 1852.54s - 1854.90s | you can come up with the same results as we do. 1854.90s - 1856.66s | And if you don't, you explain to us 1856.66s - 1858.14s | why you don't have the same results 1858.14s - 1862.22s | and maybe we have to change something on our side. 1862.22s - 1865.70s | So this is an overview of how this process works. 1865.70s - 1868.74s | So the relays have this little piece of information 1868.74s - 1871.02s | that they upload to directory authorities, 1871.02s - 1874.34s | which are special nodes in the network. 1874.34s - 1876.42s | And there is a service that's called collector 1877.38s - 1882.30s | which downloads these documents and archives them. 1882.30s - 1886.30s | And then we have a service that takes all these documents, 1886.30s - 1888.02s | parses them, puts them in a database 1888.02s - 1891.62s | and extracts time series and network statistics. 1891.62s - 1894.66s | And I said that all the data is public 1894.66s - 1897.54s | and this is true, but there are a few exceptions. 1897.54s - 1902.14s | The first exception is that we are in a race with sensors. 1902.14s - 1904.46s | And these sensors have a lot of resources 1904.46s - 1907.54s | and sometimes we need to protect some of the data 1907.54s - 1909.74s | in order to keep up with this race. 1910.82s - 1913.46s | And the other exception that we do 1913.46s - 1915.66s | is that sometimes we don't publish data 1915.66s - 1918.14s | when it can put user at risk. 1918.14s - 1920.26s | For example, if we have a specific country 1920.26s - 1922.90s | which has very few Tor users, 1922.90s - 1925.46s | and we believe that publishing those numbers 1925.46s - 1928.30s | can help somebody else that has other data 1928.30s - 1929.58s | to anonymize those users, 1929.58s - 1931.66s | so we don't publish that numbers. 1931.66s - 1937.28s | And I think the threshold is about 100 users. 1937.28s - 1941.28s | So we've been talking about metrics 1941.28s - 1943.32s |


chunk_id: 214_chunk_40
chunk_text:
And I said that all the data is public 1894.66s - 1897.54s | and this is true, but there are a few exceptions. 1897.54s - 1902.14s | The first exception is that we are in a race with sensors. 1902.14s - 1904.46s | And these sensors have a lot of resources 1904.46s - 1907.54s | and sometimes we need to protect some of the data 1907.54s - 1909.74s | in order to keep up with this race. 1910.82s - 1913.46s | And the other exception that we do 1913.46s - 1915.66s | is that sometimes we don't publish data 1915.66s - 1918.14s | when it can put user at risk. 1918.14s - 1920.26s | For example, if we have a specific country 1920.26s - 1922.90s | which has very few Tor users, 1922.90s - 1925.46s | and we believe that publishing those numbers 1925.46s - 1928.30s | can help somebody else that has other data 1928.30s - 1929.58s | to anonymize those users, 1929.58s - 1931.66s | so we don't publish that numbers. 1931.66s - 1937.28s | And I think the threshold is about 100 users. 1937.28s - 1941.28s | So we've been talking about metrics 1941.28s - 1943.32s | for understanding the status of the network, 1943.32s - 1945.04s | but also we have other metrics 1945.04s - 1948.80s | that helps us understand if a node is misbehaving 1948.80s - 1952.72s | and also give us an idea of how the network is performing 1952.72s - 1955.08s | from a point of view of performances 1955.08s - 1958.28s | and bandwidth and so on. 1958.28s - 1961.12s | So we look at traffic users patterns, 1961.12s - 1963.96s | we have actual bandwidth measurements 1964.64s - 1966.44s | and we have performance measurements. 1967.72s - 1970.96s | So one thing that we do is bandwidth probing 1970.96s - 1972.72s | and we have a service, 1972.72s - 1974.08s | we actually have a bunch of service 1974.08s - 1976.08s | that are called bandwidth authorities. 1976.08s - 1979.60s | And what they do is that they create a two hop circuit 1979.60s - 1981.12s | with the measured relay, 1981.12s - 1983.76s | connect to a web server, download the files 1983.76s - 1986.40s | and gives us an idea of the time it takes 1986.40s - 1988.44s | to download that file. 1988.44s - 1991.80s | And


chunk_id: 214_chunk_41
chunk_text:
for understanding the status of the network, 1943.32s - 1945.04s | but also we have other metrics 1945.04s - 1948.80s | that helps us understand if a node is misbehaving 1948.80s - 1952.72s | and also give us an idea of how the network is performing 1952.72s - 1955.08s | from a point of view of performances 1955.08s - 1958.28s | and bandwidth and so on. 1958.28s - 1961.12s | So we look at traffic users patterns, 1961.12s - 1963.96s | we have actual bandwidth measurements 1964.64s - 1966.44s | and we have performance measurements. 1967.72s - 1970.96s | So one thing that we do is bandwidth probing 1970.96s - 1972.72s | and we have a service, 1972.72s - 1974.08s | we actually have a bunch of service 1974.08s - 1976.08s | that are called bandwidth authorities. 1976.08s - 1979.60s | And what they do is that they create a two hop circuit 1979.60s - 1981.12s | with the measured relay, 1981.12s - 1983.76s | connect to a web server, download the files 1983.76s - 1986.40s | and gives us an idea of the time it takes 1986.40s - 1988.44s | to download that file. 1988.44s - 1991.80s | And we can compare this with the bandwidth 1991.80s - 1993.96s | that actually the relays declares 1993.96s - 1997.32s | and when they send their descriptors file 1997.32s - 1998.64s | to the directory authorities. 1998.64s - 2000.64s | And in certain case we can find out 2000.64s - 2001.84s | if some relays, for example, 2001.84s - 2004.52s | is trying to game the system and tells us 2004.52s - 2009.19s | that they have more bandwidth than they actually do. 2009.19s - 2012.39s | And another thing that we do is performance measurements 2012.39s - 2014.79s | and we do them for two reasons. 2014.79s - 2017.63s | One of them is we want to know how the network performs 2017.63s - 2019.59s | from different points in the world. 2019.59s - 2023.47s | For example, we want to know how the location 2024.35s - 2026.91s | of a user compared to the location of relays distribution 2026.91s - 2031.31s | globally affects the bandwidth, 2031.31s - 2034.03s | affect other measurements. 2034.03s - 2035.99s | And the other thing is that we want to know 2035.99s - 2040.39s | for censored regions how the network performs 2040.39s


chunk_id: 214_chunk_42
chunk_text:
we can compare this with the bandwidth 1991.80s - 1993.96s | that actually the relays declares 1993.96s - 1997.32s | and when they send their descriptors file 1997.32s - 1998.64s | to the directory authorities. 1998.64s - 2000.64s | And in certain case we can find out 2000.64s - 2001.84s | if some relays, for example, 2001.84s - 2004.52s | is trying to game the system and tells us 2004.52s - 2009.19s | that they have more bandwidth than they actually do. 2009.19s - 2012.39s | And another thing that we do is performance measurements 2012.39s - 2014.79s | and we do them for two reasons. 2014.79s - 2017.63s | One of them is we want to know how the network performs 2017.63s - 2019.59s | from different points in the world. 2019.59s - 2023.47s | For example, we want to know how the location 2024.35s - 2026.91s | of a user compared to the location of relays distribution 2026.91s - 2031.31s | globally affects the bandwidth, 2031.31s - 2034.03s | affect other measurements. 2034.03s - 2035.99s | And the other thing is that we want to know 2035.99s - 2040.39s | for censored regions how the network performs 2040.39s - 2043.71s | for those users in those extreme situation. 2043.71s - 2047.55s | So the way this works is that we have a Tor client 2047.55s - 2049.99s | and a web server and then we build a server, 2049.99s - 2052.03s | a circuit directly to the web server, 2052.03s - 2054.99s | download again a file and we measure the bandwidth, 2054.99s - 2057.23s | we measure the time it takes to build the circuit 2057.23s - 2063.46s | and if there are errors, for example. 2063.46s - 2068.14s | And so with all this data, we know it's open, 2068.14s - 2069.22s | we know it's accessible, 2069.22s - 2072.70s | we know everyone can start learning about Tor 2072.70s - 2077.66s | and basically it's documented, it's data, 2077.66s - 2079.94s | but we also know there is a learning curve. 2079.94s - 2083.02s | We also know that you need resources sometimes 2083.02s - 2086.62s | to store this data, you need time to parse it. 2086.62s - 2091.62s | And so we decided to open a little bit more 2092.38s - 2095.06s | the way people can access metrics data. 2095.06s - 2098.02s


chunk_id: 214_chunk_43
chunk_text:
- 2043.71s | for those users in those extreme situation. 2043.71s - 2047.55s | So the way this works is that we have a Tor client 2047.55s - 2049.99s | and a web server and then we build a server, 2049.99s - 2052.03s | a circuit directly to the web server, 2052.03s - 2054.99s | download again a file and we measure the bandwidth, 2054.99s - 2057.23s | we measure the time it takes to build the circuit 2057.23s - 2063.46s | and if there are errors, for example. 2063.46s - 2068.14s | And so with all this data, we know it's open, 2068.14s - 2069.22s | we know it's accessible, 2069.22s - 2072.70s | we know everyone can start learning about Tor 2072.70s - 2077.66s | and basically it's documented, it's data, 2077.66s - 2079.94s | but we also know there is a learning curve. 2079.94s - 2083.02s | We also know that you need resources sometimes 2083.02s - 2086.62s | to store this data, you need time to parse it. 2086.62s - 2091.62s | And so we decided to open a little bit more 2092.38s - 2095.06s | the way people can access metrics data. 2095.06s - 2098.02s | We are working on an API which is already embedded, 2098.02s - 2099.78s | it's already accessible. 2099.78s - 2103.50s | And we hope that this API can lower the barrier 2103.50s - 2107.02s | for researchers to start working with metrics data. 2107.02s - 2108.90s | We have few endpoints available, 2108.90s - 2111.86s | more coming up soon as we are developing. 2111.86s - 2115.34s | So you can access metrics-api.torproject.org, 2115.34s - 2117.54s | the two endpoints that are available are details 2117.54s - 2121.74s | and summary and give you some information of relays. 2121.74s - 2123.78s | I forgot to mention the API is called 2123.78s - 2127.54s | the Network Status API or NSA for short. 2127.54s - 2132.54s | So one nice thing is that besides that you're querying 2135.14s - 2138.02s | the NSA to get the information, 2138.02s - 2141.10s | is that it has a little bit more flexibility 2141.10s - 2142.54s | compared to parsing the data 2142.54s - 2144.78s | or what we are offering right now. 2144.78s - 2148.86s | And is that you can do, for example, historical queries. 2148.86s - 2151.26s | So there


chunk_id: 214_chunk_44
chunk_text:
| We are working on an API which is already embedded, 2098.02s - 2099.78s | it's already accessible. 2099.78s - 2103.50s | And we hope that this API can lower the barrier 2103.50s - 2107.02s | for researchers to start working with metrics data. 2107.02s - 2108.90s | We have few endpoints available, 2108.90s - 2111.86s | more coming up soon as we are developing. 2111.86s - 2115.34s | So you can access metrics-api.torproject.org, 2115.34s - 2117.54s | the two endpoints that are available are details 2117.54s - 2121.74s | and summary and give you some information of relays. 2121.74s - 2123.78s | I forgot to mention the API is called 2123.78s - 2127.54s | the Network Status API or NSA for short. 2127.54s - 2132.54s | So one nice thing is that besides that you're querying 2135.14s - 2138.02s | the NSA to get the information, 2138.02s - 2141.10s | is that it has a little bit more flexibility 2141.10s - 2142.54s | compared to parsing the data 2142.54s - 2144.78s | or what we are offering right now. 2144.78s - 2148.86s | And is that you can do, for example, historical queries. 2148.86s - 2151.26s | So there is a published variable 2151.26s - 2153.78s | that you can parse in the URL. 2153.78s - 2158.78s | And you parse, you send a date, just year, month, day, 2158.94s - 2161.50s | and then you get all the relays and bridges 2161.50s - 2164.38s | that were in the network within a week of the parcel date. 2164.38s - 2168.94s | So this is something new and that I believe 2168.94s - 2172.18s | can give a lot of flexibility to researchers 2172.18s - 2173.86s | to start experimenting. 2173.86s - 2175.82s | I hope it helps relay operators 2175.82s - 2178.22s | checking the status of their nodes. 2178.22s - 2180.06s | And you can check out the documentation. 2180.06s - 2184.30s | We welcome feedback and bugs reports, of course, 2184.30s - 2186.72s | because expect it to be broken. 2189.34s - 2193.22s | And yeah, I think it's part of the Tor Project mission 2193.22s - 2196.32s | to create and develop anonymity and privacy technology, 2196.36s - 2199.28s | but also to further its understanding, 2199.28s - 2202.16s | scientific understanding, popular understanding. 2202.16s - 2208.43s | And this is one step forward in


chunk_id: 214_chunk_45
chunk_text:
is a published variable 2151.26s - 2153.78s | that you can parse in the URL. 2153.78s - 2158.78s | And you parse, you send a date, just year, month, day, 2158.94s - 2161.50s | and then you get all the relays and bridges 2161.50s - 2164.38s | that were in the network within a week of the parcel date. 2164.38s - 2168.94s | So this is something new and that I believe 2168.94s - 2172.18s | can give a lot of flexibility to researchers 2172.18s - 2173.86s | to start experimenting. 2173.86s - 2175.82s | I hope it helps relay operators 2175.82s - 2178.22s | checking the status of their nodes. 2178.22s - 2180.06s | And you can check out the documentation. 2180.06s - 2184.30s | We welcome feedback and bugs reports, of course, 2184.30s - 2186.72s | because expect it to be broken. 2189.34s - 2193.22s | And yeah, I think it's part of the Tor Project mission 2193.22s - 2196.32s | to create and develop anonymity and privacy technology, 2196.36s - 2199.28s | but also to further its understanding, 2199.28s - 2202.16s | scientific understanding, popular understanding. 2202.16s - 2208.43s | And this is one step forward in that direction. 2208.43s - 2211.87s | Okay, so we've learned a bit about the relay community side 2211.87s - 2214.35s | and we've learned a bit about the metrics 2214.35s - 2216.31s | and network measurement side. 2216.31s - 2218.23s | What can you do to help us? 2218.23s - 2222.15s | So one big piece is get involved in the relay community 2222.15s - 2224.99s | and run a relay around the world at your university, 2224.99s - 2229.75s | at your work, at your VPS, wherever you happen to be. 2229.75s - 2231.83s | Maybe there are sysadmins in here right now 2231.83s - 2234.07s | who would love to add an extra relay 2234.07s - 2236.59s | on their volunteer rack. 2236.59s - 2239.35s | And so part of that is the more relays we have, 2239.35s - 2241.03s | the more performance we can handle, 2241.03s - 2243.99s | the more users we can give good performance to. 2243.99s - 2246.77s | And the other piece of it is the more relays we have 2246.77s - 2250.35s | run by good people, the smaller the impact is 2250.35s - 2253.35s | when some bad person shows


chunk_id: 214_chunk_46
chunk_text:
that direction. 2208.43s - 2211.87s | Okay, so we've learned a bit about the relay community side 2211.87s - 2214.35s | and we've learned a bit about the metrics 2214.35s - 2216.31s | and network measurement side. 2216.31s - 2218.23s | What can you do to help us? 2218.23s - 2222.15s | So one big piece is get involved in the relay community 2222.15s - 2224.99s | and run a relay around the world at your university, 2224.99s - 2229.75s | at your work, at your VPS, wherever you happen to be. 2229.75s - 2231.83s | Maybe there are sysadmins in here right now 2231.83s - 2234.07s | who would love to add an extra relay 2234.07s - 2236.59s | on their volunteer rack. 2236.59s - 2239.35s | And so part of that is the more relays we have, 2239.35s - 2241.03s | the more performance we can handle, 2241.03s - 2243.99s | the more users we can give good performance to. 2243.99s - 2246.77s | And the other piece of it is the more relays we have 2246.77s - 2250.35s | run by good people, the smaller the impact is 2250.35s - 2253.35s | when some bad person shows up to try to run a relay. 2253.39s - 2256.75s | So the way that we dilute the people trying 2256.75s - 2260.51s | to do the attacks is we get more and more hacker spaces 2260.51s - 2263.87s | and individuals and hackers and volunteers 2263.87s - 2266.03s | to be participating in this community. 2266.03s - 2267.87s | And yes, location matters. 2267.87s - 2270.39s | If you can run a relay that's not in Germany 2270.39s - 2273.55s | or the Netherlands, then you'll be showing up 2273.55s - 2275.11s | in a part of the Tor network 2275.11s - 2277.67s | that doesn't have as many relays right now. 2277.67s - 2280.97s | So for example, the U.S. is nowhere near the top 2280.97s - 2283.01s | in terms of countries that are contributing 2283.67s - 2284.51s | to the Tor network. 2284.51s - 2285.77s | I hear the U.S. has lots of hackers 2285.77s - 2289.47s | and lots of internet connections and lots of computers. 2289.47s - 2292.55s | So there's certainly an opportunity for you 2292.55s - 2295.63s | to help us make the U.S. more impactful, 2295.63s - 2297.83s | which in


chunk_id: 214_chunk_47
chunk_text:
up to try to run a relay. 2253.39s - 2256.75s | So the way that we dilute the people trying 2256.75s - 2260.51s | to do the attacks is we get more and more hacker spaces 2260.51s - 2263.87s | and individuals and hackers and volunteers 2263.87s - 2266.03s | to be participating in this community. 2266.03s - 2267.87s | And yes, location matters. 2267.87s - 2270.39s | If you can run a relay that's not in Germany 2270.39s - 2273.55s | or the Netherlands, then you'll be showing up 2273.55s - 2275.11s | in a part of the Tor network 2275.11s - 2277.67s | that doesn't have as many relays right now. 2277.67s - 2280.97s | So for example, the U.S. is nowhere near the top 2280.97s - 2283.01s | in terms of countries that are contributing 2283.67s - 2284.51s | to the Tor network. 2284.51s - 2285.77s | I hear the U.S. has lots of hackers 2285.77s - 2289.47s | and lots of internet connections and lots of computers. 2289.47s - 2292.55s | So there's certainly an opportunity for you 2292.55s - 2295.63s | to help us make the U.S. more impactful, 2295.63s - 2297.83s | which in turn makes performance better 2297.83s - 2299.83s | for users in this hemisphere. 2299.83s - 2302.61s | So think about diversity of location, 2302.61s - 2304.67s | but wherever you can run it is great. 2304.67s - 2306.19s | Please add more relays. 2306.19s - 2307.87s | And also I'm happy to chat with you more 2307.87s - 2312.15s | about exit policies and whether you'll end up 2312.17s - 2315.37s | in IP block lists and so on after this. 2316.41s - 2319.97s | And the other side of that is community.torproject.org 2319.97s - 2323.63s | has a lot of how-tos and guides on, 2323.63s - 2325.97s | I want to set up a bridge on Debian. 2325.97s - 2327.01s | How do I do it? 2327.01s - 2328.13s | Here are the steps. 2328.13s - 2329.57s | I want to do a snowflake. 2329.57s - 2330.65s | Here's the steps. 2330.65s - 2333.19s | I want to go to a relay operator meetup 2333.19s - 2335.65s | or learn about the relay associations. 2335.65s - 2338.03s | Here are the lists and documentation. 2338.03s - 2339.39s | So there's a lot there. 2339.39s - 2341.09s | And join


chunk_id: 214_chunk_48
chunk_text:
turn makes performance better 2297.83s - 2299.83s | for users in this hemisphere. 2299.83s - 2302.61s | So think about diversity of location, 2302.61s - 2304.67s | but wherever you can run it is great. 2304.67s - 2306.19s | Please add more relays. 2306.19s - 2307.87s | And also I'm happy to chat with you more 2307.87s - 2312.15s | about exit policies and whether you'll end up 2312.17s - 2315.37s | in IP block lists and so on after this. 2316.41s - 2319.97s | And the other side of that is community.torproject.org 2319.97s - 2323.63s | has a lot of how-tos and guides on, 2323.63s - 2325.97s | I want to set up a bridge on Debian. 2325.97s - 2327.01s | How do I do it? 2327.01s - 2328.13s | Here are the steps. 2328.13s - 2329.57s | I want to do a snowflake. 2329.57s - 2330.65s | Here's the steps. 2330.65s - 2333.19s | I want to go to a relay operator meetup 2333.19s - 2335.65s | or learn about the relay associations. 2335.65s - 2338.03s | Here are the lists and documentation. 2338.03s - 2339.39s | So there's a lot there. 2339.39s - 2341.09s | And join the community in general. 2341.09s - 2342.83s | Come to the relay operator meetups. 2342.83s - 2346.43s | We don't have a formal official relay operator meetup 2346.43s - 2347.99s | here at DEF CON this year 2347.99s - 2351.13s | because this is one enormous room 2351.13s - 2353.43s | and we didn't figure out how to do the spaces. 2353.43s - 2356.59s | Maybe next year we'll join Privacy Village or something. 2356.59s - 2359.15s | But you can come talk to us afterwards 2359.15s - 2361.79s | and we'd love to chat with you more about relays. 2361.79s - 2363.35s | And if you can't run a relay, 2363.35s - 2370.55s | run a bridge or a snowflake or contribute in other ways. 2370.57s - 2372.21s | Yeah, so you can also get involved 2372.21s - 2374.37s | with the network health team. 2374.37s - 2376.65s | If you're interested in network data analysis, 2376.65s - 2379.57s | hunting for bad relays, there's certainly space for that. 2379.57s - 2382.31s | We hope the API can help you. 2382.31s - 2384.21s | There is also one thing that we do 2384.21s - 2387.33s | which is creating


chunk_id: 214_chunk_49
chunk_text:
the community in general. 2341.09s - 2342.83s | Come to the relay operator meetups. 2342.83s - 2346.43s | We don't have a formal official relay operator meetup 2346.43s - 2347.99s | here at DEF CON this year 2347.99s - 2351.13s | because this is one enormous room 2351.13s - 2353.43s | and we didn't figure out how to do the spaces. 2353.43s - 2356.59s | Maybe next year we'll join Privacy Village or something. 2356.59s - 2359.15s | But you can come talk to us afterwards 2359.15s - 2361.79s | and we'd love to chat with you more about relays. 2361.79s - 2363.35s | And if you can't run a relay, 2363.35s - 2370.55s | run a bridge or a snowflake or contribute in other ways. 2370.57s - 2372.21s | Yeah, so you can also get involved 2372.21s - 2374.37s | with the network health team. 2374.37s - 2376.65s | If you're interested in network data analysis, 2376.65s - 2379.57s | hunting for bad relays, there's certainly space for that. 2379.57s - 2382.31s | We hope the API can help you. 2382.31s - 2384.21s | There is also one thing that we do 2384.21s - 2387.33s | which is creating tools for relay operators. 2387.33s - 2388.89s | For example, Tor Weather, 2388.89s - 2391.09s | which gives a notification to operators 2391.09s - 2393.61s | when something is not right with the relay. 2393.61s - 2395.81s | But also you can get involved with 2395.81s - 2400.59s | for different projects, the network status API, 2400.59s - 2405.59s | the parser that we use to parse the network documents in Java 2405.59s - 2408.07s | and also we are trying to work on 2408.07s - 2412.91s | a better way to visualize network data. 2412.91s - 2417.31s | We are trying to move the parser from Java to Rust. 2417.31s - 2420.69s | There is a link to our wiki, please get involved. 2420.69s - 2423.35s | And if you're thinking to contribute to Tor 2423.35s - 2425.19s | in any possible way, 2425.19s - 2427.05s | you're helping me, you're helping Roger, 2427.05s - 2429.35s | you're helping our colleagues, 2429.35s - 2431.49s | you're helping us fighting the good fight 2431.49s - 2445.07s | for privacy, anonymity and internet freedom. 2445.07s - 2445.95s | Yeah, excellent. 2445.95s - 2449.05s | So we've told you a bit about


chunk_id: 214_chunk_50
chunk_text:
tools for relay operators. 2387.33s - 2388.89s | For example, Tor Weather, 2388.89s - 2391.09s | which gives a notification to operators 2391.09s - 2393.61s | when something is not right with the relay. 2393.61s - 2395.81s | But also you can get involved with 2395.81s - 2400.59s | for different projects, the network status API, 2400.59s - 2405.59s | the parser that we use to parse the network documents in Java 2405.59s - 2408.07s | and also we are trying to work on 2408.07s - 2412.91s | a better way to visualize network data. 2412.91s - 2417.31s | We are trying to move the parser from Java to Rust. 2417.31s - 2420.69s | There is a link to our wiki, please get involved. 2420.69s - 2423.35s | And if you're thinking to contribute to Tor 2423.35s - 2425.19s | in any possible way, 2425.19s - 2427.05s | you're helping me, you're helping Roger, 2427.05s - 2429.35s | you're helping our colleagues, 2429.35s - 2431.49s | you're helping us fighting the good fight 2431.49s - 2445.07s | for privacy, anonymity and internet freedom. 2445.07s - 2445.95s | Yeah, excellent. 2445.95s - 2449.05s | So we've told you a bit about how you can help us, 2449.05s - 2450.97s | but it is about helping us 2450.97s - 2453.73s | and it's also about helping all of the activists 2453.73s - 2455.07s | and people around the world, 2455.07s - 2457.73s | especially the people who don't have the power 2457.73s - 2460.93s | to have a military or bodyguards or whatever, 2460.93s - 2463.05s | yet they want to change their society. 2463.05s - 2466.79s | So we will be doing an Ask Us Anything You Like 2466.79s - 2470.27s | at the Tor booth in the vendor area right after this. 2470.27s - 2473.03s | We've got these cool badges and other things available 2473.03s - 2475.83s | or you can just ask us every possible Tor question. 2475.83s - 2477.77s | We will be there until you have run out 2477.77s - 2479.27s | of all of your Tor questions 2479.27s - 2481.25s | and Argoon has kindly volunteered 2481.25s - 2483.79s | to escort all of us over there 2483.79s - 2486.03s | and we'll hang out as long as you would like 2486.03s - 2487.55s | to answer all your questions. 2487.55s - 2488.39s | Thank


chunk_id: 214_chunk_51
chunk_text:
how you can help us, 2449.05s - 2450.97s | but it is about helping us 2450.97s - 2453.73s | and it's also about helping all of the activists 2453.73s - 2455.07s | and people around the world, 2455.07s - 2457.73s | especially the people who don't have the power 2457.73s - 2460.93s | to have a military or bodyguards or whatever, 2460.93s - 2463.05s | yet they want to change their society. 2463.05s - 2466.79s | So we will be doing an Ask Us Anything You Like 2466.79s - 2470.27s | at the Tor booth in the vendor area right after this. 2470.27s - 2473.03s | We've got these cool badges and other things available 2473.03s - 2475.83s | or you can just ask us every possible Tor question. 2475.83s - 2477.77s | We will be there until you have run out 2477.77s - 2479.27s | of all of your Tor questions 2479.27s - 2481.25s | and Argoon has kindly volunteered 2481.25s - 2483.79s | to escort all of us over there 2483.79s - 2486.03s | and we'll hang out as long as you would like 2486.03s - 2487.55s | to answer all your questions. 2487.55s - 2488.39s | Thank you.


chunk_id: 215_chunk_0
chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=dldX9UFhNTs", "title": "DEF CON 32 - Laundering Money - Michael Orlitzky", "description": "CSC ServiceWorks is a large vendor of pay-to-play laundry machines in apartments and condomiums. Most are Speed Queens, but newer CSC-branded machines use an app for payment and have custom circuitry inside. Many however accept quarters as well. We show that, when all else fails, you can always physically bypass the coin slot to run the machines for free.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1173, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.24s - 5.88s | This text was transcribed using whisper model: large-v2 Alright. So, uh, my name is Michael, and this is the title of my talk. It is Laundering 5.88s - 10.08s | Money. If you're here to learn about laundering money, you're in the wrong place. Because 10.08s - 15.08s | this talk is laundering money. Um, what we're going to do today is do some laundry for free 18.02s - 23.02s | on machines that would really rather that you pay for them. Uh, in other words, what we're 23.02s - 28.02s | going to do is save, that's right, laundering money. And from whom are we going to save? 31.16s - 37.60s | A company called CSC ServiceWorks, who is the villain of this talk. Uh, and that is despite 37.60s - 42.62s | the fact that, uh, to the untrained eye it's going to look like I am the one doing all of the 44.32s - 49.32s | bad things. Uh, but I promise you, uh, I know we are the protagonists. Um, this is our 51.52s - 56.52s | target. It is, uh, these are the washers in my building. Uh, they are CSC branded. Um, and 57.48s - 63.94s | these are the dryers on the other side of the hallway. Uh, they are speed queens. Speed 63.94s - 69.42s | queens are the most popular model of commercial washer and dryer in the United States. Uh, and 69.42s - 74.56s | I made that up, but it's probably true. Uh, the manuals for these can all be found online. 74.56s - 79.10s | They're extensively documented. And the manuals tell you how to hack the machine. But the 79.10s - 85.14s | problem is that CSC, because they suck, have modified the base speed queens so that nothing 85.16s - 91.52s | works. Right? So, here's a comprehensive


chunk_id: 215_chunk_1
chunk_text:
| A company called CSC ServiceWorks, who is the villain of this talk. Uh, and that is despite 37.60s - 42.62s | the fact that, uh, to the untrained eye it's going to look like I am the one doing all of the 44.32s - 49.32s | bad things. Uh, but I promise you, uh, I know we are the protagonists. Um, this is our 51.52s - 56.52s | target. It is, uh, these are the washers in my building. Uh, they are CSC branded. Um, and 57.48s - 63.94s | these are the dryers on the other side of the hallway. Uh, they are speed queens. Speed 63.94s - 69.42s | queens are the most popular model of commercial washer and dryer in the United States. Uh, and 69.42s - 74.56s | I made that up, but it's probably true. Uh, the manuals for these can all be found online. 74.56s - 79.10s | They're extensively documented. And the manuals tell you how to hack the machine. But the 79.10s - 85.14s | problem is that CSC, because they suck, have modified the base speed queens so that nothing 85.16s - 91.52s | works. Right? So, here's a comprehensive list of things that do not work. And I have to do 91.52s - 96.26s | this because if I don't, inevitably someone is going to come up to me at the end of the talk 96.26s - 101.44s | and tell me that I'm an idiot and that I've wasted my time and that he saw a TikTok video and 101.44s - 105.88s | all you have to do is press two buttons and the machine will do whatever you want. It's not 105.88s - 110.58s | true. Uh, so we're going to start with the stupidest tricks. Who played video games in the 110.58s - 116.10s | 80s or 90s? Uh, you used to be able to go to an arcade, uh, and then go to the hardware store, 116.10s - 119.90s | buy a washer that was just quarter sized and put it in the machine and it would give you 119.90s - 125.30s | credit for a quarter. Uh, these days the washers cost more than the quarters do, but it's the 125.30s - 129.70s | principle. There's a type of person who is willing to spend 40 cents to make sure that the 129.70s -


chunk_id: 215_chunk_2
chunk_text:
list of things that do not work. And I have to do 91.52s - 96.26s | this because if I don't, inevitably someone is going to come up to me at the end of the talk 96.26s - 101.44s | and tell me that I'm an idiot and that I've wasted my time and that he saw a TikTok video and 101.44s - 105.88s | all you have to do is press two buttons and the machine will do whatever you want. It's not 105.88s - 110.58s | true. Uh, so we're going to start with the stupidest tricks. Who played video games in the 110.58s - 116.10s | 80s or 90s? Uh, you used to be able to go to an arcade, uh, and then go to the hardware store, 116.10s - 119.90s | buy a washer that was just quarter sized and put it in the machine and it would give you 119.90s - 125.30s | credit for a quarter. Uh, these days the washers cost more than the quarters do, but it's the 125.30s - 129.70s | principle. There's a type of person who is willing to spend 40 cents to make sure that the 129.70s - 135.84s | other guy gets zero cents and it's nice to meet you too. Uh, the second bullet point here is 135.84s - 140.78s | if the first one fails, uh, if it knows that the washer is not a quarter, you can drill a tiny 140.78s - 145.38s | hole in a real quarter, tie a string around it and go fishing like Wile E. Coyote and just pull 145.38s - 149.28s | the quarter back out. Doesn't work. These machines have trap doors in the coin slot 149.28s - 155.46s | mechanism. So, manual mode. The two button trick from TikTok puts the machine into 155.46s - 160.34s | something called manual mode which believe it or not is documented in the manual. Uh, there 160.34s - 165.10s | are several ways to get into manual mode which gives you admin privileges and none of them 165.10s - 171.06s | work because CSC has changed the base model, uh, and all the circuitry. Uh, there's also a 171.06s - 176.88s | dip switch to turn off the entire payment system. Doesn't work. And, uh, I'm from Baltimore 176.88s - 182.14s | and there's a guy up the street,


chunk_id: 215_chunk_3
chunk_text:
135.84s | other guy gets zero cents and it's nice to meet you too. Uh, the second bullet point here is 135.84s - 140.78s | if the first one fails, uh, if it knows that the washer is not a quarter, you can drill a tiny 140.78s - 145.38s | hole in a real quarter, tie a string around it and go fishing like Wile E. Coyote and just pull 145.38s - 149.28s | the quarter back out. Doesn't work. These machines have trap doors in the coin slot 149.28s - 155.46s | mechanism. So, manual mode. The two button trick from TikTok puts the machine into 155.46s - 160.34s | something called manual mode which believe it or not is documented in the manual. Uh, there 160.34s - 165.10s | are several ways to get into manual mode which gives you admin privileges and none of them 165.10s - 171.06s | work because CSC has changed the base model, uh, and all the circuitry. Uh, there's also a 171.06s - 176.88s | dip switch to turn off the entire payment system. Doesn't work. And, uh, I'm from Baltimore 176.88s - 182.14s | and there's a guy up the street, uh, in Philadelphia who has a video on YouTube, uh, where he 182.14s - 186.48s | just takes a pocket knife basically and shorts out these two wires. This doesn't work on my 186.48s - 193.06s | model because my model doesn't have those wires. Alright, but who is CSC ServiceWorks? 193.08s - 197.46s | Well, what do they do? They take these laundry machines, they outfit them with a pay module 197.46s - 202.42s | and they put them into condo buildings and apartments and the residents pay to use them and 202.42s - 207.44s | in exchange CSC is supposed to maintain the machines. There are three ways to pay. Uh, the 207.44s - 212.00s | first is a refillable chip card, the second are these two stupid apps, and the third is with 212.00s - 217.48s | quarters, the old fashioned way, right? But, a picture is worth a thousand words. And so 217.50s - 222.64s | here's a picture of what CSC actually does. Uh, I hope you can see that it's quite 222.64s - 227.84s | impressive because it's not possible to give a lower rating than one star and there are 227.84s - 234.66s


chunk_id: 215_chunk_4
chunk_text:
uh, in Philadelphia who has a video on YouTube, uh, where he 182.14s - 186.48s | just takes a pocket knife basically and shorts out these two wires. This doesn't work on my 186.48s - 193.06s | model because my model doesn't have those wires. Alright, but who is CSC ServiceWorks? 193.08s - 197.46s | Well, what do they do? They take these laundry machines, they outfit them with a pay module 197.46s - 202.42s | and they put them into condo buildings and apartments and the residents pay to use them and 202.42s - 207.44s | in exchange CSC is supposed to maintain the machines. There are three ways to pay. Uh, the 207.44s - 212.00s | first is a refillable chip card, the second are these two stupid apps, and the third is with 212.00s - 217.48s | quarters, the old fashioned way, right? But, a picture is worth a thousand words. And so 217.50s - 222.64s | here's a picture of what CSC actually does. Uh, I hope you can see that it's quite 222.64s - 227.84s | impressive because it's not possible to give a lower rating than one star and there are 227.84s - 234.66s | 343 reviews up there. Uh, but right, the better business bureau sucks, so what if you trust 234.66s - 240.22s | Yelp? Ooh, we're, we're getting better at least. Uh, but Yelp has been known to manipulate 240.22s - 246.56s | ratings, so what if we ask Google? Yikes. Okay, uh, well this is unfair because what is a 246.58s - 253.36s | one star rating even mean? We need to be able to put this into context somehow, uh, to really, 253.36s - 258.56s | you know, be able to judge what one star means. I'm from Baltimore, uh, so to put this in 258.56s - 264.30s | context for me, I decided to find out what former residents think of the Baltimore jail after 264.30s - 270.54s | they've been released. You heard it here first, folks. Uh, having any sort of business 270.54s - 276.14s | relationship with CSC service works is less fun than jail. And not just any jail, Baltimore 276.20s - 283.20s | jail. Alright. Now, yes, a picture is worth a thousand words, but words are also worth words. 283.20s - 287.70s | Uh, specifically each word is worth one word. So


chunk_id: 215_chunk_5
chunk_text:
| 343 reviews up there. Uh, but right, the better business bureau sucks, so what if you trust 234.66s - 240.22s | Yelp? Ooh, we're, we're getting better at least. Uh, but Yelp has been known to manipulate 240.22s - 246.56s | ratings, so what if we ask Google? Yikes. Okay, uh, well this is unfair because what is a 246.58s - 253.36s | one star rating even mean? We need to be able to put this into context somehow, uh, to really, 253.36s - 258.56s | you know, be able to judge what one star means. I'm from Baltimore, uh, so to put this in 258.56s - 264.30s | context for me, I decided to find out what former residents think of the Baltimore jail after 264.30s - 270.54s | they've been released. You heard it here first, folks. Uh, having any sort of business 270.54s - 276.14s | relationship with CSC service works is less fun than jail. And not just any jail, Baltimore 276.20s - 283.20s | jail. Alright. Now, yes, a picture is worth a thousand words, but words are also worth words. 283.20s - 287.70s | Uh, specifically each word is worth one word. So I'm going to use some words to tell you 287.70s - 292.74s | about the dark patterns that CSC uses to steal money from its customers. And for two 292.74s - 296.74s | reasons. Because, right, in a minute I'm going to rob these guys and I don't want anyone to 296.74s - 301.78s | feel guilty. Uh, that's the first reason. Uh, but the second reason is that this is a 301.82s - 306.60s | security-minded audience and some of these are actually kind of good and would be funny if it 306.60s - 311.76s | wasn't my money they're stealing. The first dark pattern is with the, um, refillable chip 311.76s - 316.64s | cards, which have the same downsides that any, like, gift card have, right? Uh, if the 316.64s - 321.04s | company stops supporting them or if they change the machines, if you lose it, if it's, you 321.04s - 326.04s | know, you're fucked. Uh, these specific card machines, they have, uh, a separate refill 326.42s - 333.28s | machine and some fine print in the lower right hand corner of the refill machine that says 333.28s - 337.96s | that they do not


chunk_id: 215_chunk_6
chunk_text:
I'm going to use some words to tell you 287.70s - 292.74s | about the dark patterns that CSC uses to steal money from its customers. And for two 292.74s - 296.74s | reasons. Because, right, in a minute I'm going to rob these guys and I don't want anyone to 296.74s - 301.78s | feel guilty. Uh, that's the first reason. Uh, but the second reason is that this is a 301.82s - 306.60s | security-minded audience and some of these are actually kind of good and would be funny if it 306.60s - 311.76s | wasn't my money they're stealing. The first dark pattern is with the, um, refillable chip 311.76s - 316.64s | cards, which have the same downsides that any, like, gift card have, right? Uh, if the 316.64s - 321.04s | company stops supporting them or if they change the machines, if you lose it, if it's, you 321.04s - 326.04s | know, you're fucked. Uh, these specific card machines, they have, uh, a separate refill 326.42s - 333.28s | machine and some fine print in the lower right hand corner of the refill machine that says 333.28s - 337.96s | that they do not accept one dollar bills. Oh, they accept one dollar bills. What they won't 337.96s - 343.00s | do is put one dollar on your card and you have to call customer service and spend an hour on 343.00s - 347.76s | the phone to get your one dollar back. Uh, the third bullet point is one of my favorites. So 347.76s - 352.60s | CSC has to coordinate with the people who own the buildings to do any sort of maintenance 352.60s - 356.98s | on the machines and they're upgrading from the card machines now to the app machines. So 356.98s - 360.84s | they coordinated with my building and with several other people's buildings and you know 360.84s - 365.68s | they picked a date, put up flyers, sent out emails, uh, let everyone know when the machines 365.68s - 369.92s | are going to be changed over and then two weeks ahead of schedule at like 630 in the morning 369.92s - 374.92s | they sent in a, a covert laundry ops team to steal the old card reader machines and replace 374.92s - 379.00s | them with app machines. Now why is this a dark pattern?


chunk_id: 215_chunk_7
chunk_text:
accept one dollar bills. Oh, they accept one dollar bills. What they won't 337.96s - 343.00s | do is put one dollar on your card and you have to call customer service and spend an hour on 343.00s - 347.76s | the phone to get your one dollar back. Uh, the third bullet point is one of my favorites. So 347.76s - 352.60s | CSC has to coordinate with the people who own the buildings to do any sort of maintenance 352.60s - 356.98s | on the machines and they're upgrading from the card machines now to the app machines. So 356.98s - 360.84s | they coordinated with my building and with several other people's buildings and you know 360.84s - 365.68s | they picked a date, put up flyers, sent out emails, uh, let everyone know when the machines 365.68s - 369.92s | are going to be changed over and then two weeks ahead of schedule at like 630 in the morning 369.92s - 374.92s | they sent in a, a covert laundry ops team to steal the old card reader machines and replace 374.92s - 379.00s | them with app machines. Now why is this a dark pattern? Because it strands like a hundred 379.00s - 383.14s | people with a stupid card with money on it that they can't spend that they've already paid 383.14s - 388.44s | for. Uh, and the only way to get it back now is to just snail mail the card to them and pray 388.44s - 391.94s | because you don't even know how much money is on the card because they stole the fucking card 391.94s - 398.12s | reader. Uh, the machine settings also don't work in your favor. When I originally did this 398.12s - 402.46s | work the machines in my building were a dollar eighty five a piece and you could only put ten 402.46s - 406.96s | dollars at a time on your card and if you ever tried to spend ten dollars one eighty five at a 406.96s - 412.00s | time you can't. So when you lose that card they get some of your money. Now they've since 412.00s - 416.64s | remedied this by raising the price to two dollars but also fucked them for raising the price. 416.64s - 422.74s | Um, if you put quarters into the uh coin operated


chunk_id: 215_chunk_8
chunk_text:
Because it strands like a hundred 379.00s - 383.14s | people with a stupid card with money on it that they can't spend that they've already paid 383.14s - 388.44s | for. Uh, and the only way to get it back now is to just snail mail the card to them and pray 388.44s - 391.94s | because you don't even know how much money is on the card because they stole the fucking card 391.94s - 398.12s | reader. Uh, the machine settings also don't work in your favor. When I originally did this 398.12s - 402.46s | work the machines in my building were a dollar eighty five a piece and you could only put ten 402.46s - 406.96s | dollars at a time on your card and if you ever tried to spend ten dollars one eighty five at a 406.96s - 412.00s | time you can't. So when you lose that card they get some of your money. Now they've since 412.00s - 416.64s | remedied this by raising the price to two dollars but also fucked them for raising the price. 416.64s - 422.74s | Um, if you put quarters into the uh coin operated machines you lose the ability to change any 422.74s - 428.04s | of the cycle options so if you try to change from say uh cold water to hot water and you 428.04s - 432.22s | forget after you've put a quarter in you can't. You have to take your laundry, move it next 432.22s - 436.92s | door, pay again and then run with the hot water. Alright. I don't know too much about the 436.92s - 441.88s | apps because I would rather die than install an app on my phone. Uh, but I do know that the 441.88s - 446.70s | apps will take your money at any time over cellular data but you can only redeem the credit if 446.70s - 452.26s | the building's wifi is up and if CSC servers are working. Finally, the oldest dark pattern in 452.26s - 456.46s | the world is to just have shitty customer service. Once they have your money they don't want 456.46s - 461.10s | to answer the phone and they especially don't want to give it back. Uh, they also do not fix 461.10s - 464.78s | the machines which you may recall if you


chunk_id: 215_chunk_9
chunk_text:
machines you lose the ability to change any 422.74s - 428.04s | of the cycle options so if you try to change from say uh cold water to hot water and you 428.04s - 432.22s | forget after you've put a quarter in you can't. You have to take your laundry, move it next 432.22s - 436.92s | door, pay again and then run with the hot water. Alright. I don't know too much about the 436.92s - 441.88s | apps because I would rather die than install an app on my phone. Uh, but I do know that the 441.88s - 446.70s | apps will take your money at any time over cellular data but you can only redeem the credit if 446.70s - 452.26s | the building's wifi is up and if CSC servers are working. Finally, the oldest dark pattern in 452.26s - 456.46s | the world is to just have shitty customer service. Once they have your money they don't want 456.46s - 461.10s | to answer the phone and they especially don't want to give it back. Uh, they also do not fix 461.10s - 464.78s | the machines which you may recall if you were paying attention is the one thing that they're 464.80s - 468.84s | supposed to do in exchange for your money. And the last one doesn't fit anywhere but I threw 468.84s - 475.52s | it in there. They lost a re- a Rico lawsuit for doing Rico shit. So, I've concluded the CSC's 475.52s - 479.28s | business model is just stealing quarters from old ladies because what are you going to do 479.28s - 485.30s | about it? Well, what am I going to do about it? This is me. Um, why do I care so much? So, 485.30s - 490.26s | I've been in the industry for like 20 years doing programming, system administration, 490.26s - 494.20s | hacking. I was one of the teenagers who got in trouble for taking down MySpace. These 494.22s - 499.92s | days I do more benign stuff. I, uh, file CVEs against open source projects and contribute 499.92s - 505.96s | fixes. Uh, I'm a Gen 2 Linux developer. I have a PhD in math. Doesn't matter. Nobody cares. 505.96s - 511.00s | All you need to know about me is that I really hate being taken advantage of. Especially


chunk_id: 215_chunk_10
chunk_text:
were paying attention is the one thing that they're 464.80s - 468.84s | supposed to do in exchange for your money. And the last one doesn't fit anywhere but I threw 468.84s - 475.52s | it in there. They lost a re- a Rico lawsuit for doing Rico shit. So, I've concluded the CSC's 475.52s - 479.28s | business model is just stealing quarters from old ladies because what are you going to do 479.28s - 485.30s | about it? Well, what am I going to do about it? This is me. Um, why do I care so much? So, 485.30s - 490.26s | I've been in the industry for like 20 years doing programming, system administration, 490.26s - 494.20s | hacking. I was one of the teenagers who got in trouble for taking down MySpace. These 494.22s - 499.92s | days I do more benign stuff. I, uh, file CVEs against open source projects and contribute 499.92s - 505.96s | fixes. Uh, I'm a Gen 2 Linux developer. I have a PhD in math. Doesn't matter. Nobody cares. 505.96s - 511.00s | All you need to know about me is that I really hate being taken advantage of. Especially when 511.00s - 516.20s | it's for a small amount of money. And especially when the implication is that I just won't do 516.20s - 520.44s | anything about it. Now, this is DEF CON. So, there are a lot of people here who are way 520.44s - 525.46s | better than me with electronic thing it, uh, what's it's and thingamabobs. However, I do 528.32s - 534.32s | have a very particular set of skills. So, I'm going to use those skills to give myself a 534.32s - 541.29s | refund. Why? So, first we gotta get into these things, right? These are the dryers. That's the 545.95s - 551.13s | front view. That's the top view. Those are Phillips screws. And this is a Phillips 551.13s - 556.15s | screwdriver. This is a 20 minute talk. I'm not sure what you expected. The washers are a 558.61s - 562.59s | little bit harder, right? There's a tubular lock on the front of that thing and so we've got 562.59s - 568.09s | some options. The first option, hardcore mode. If you know how to pick tubular locks, that'll 568.09s - 572.69s | get you right in. But there's


chunk_id: 215_chunk_11
chunk_text:
when 511.00s - 516.20s | it's for a small amount of money. And especially when the implication is that I just won't do 516.20s - 520.44s | anything about it. Now, this is DEF CON. So, there are a lot of people here who are way 520.44s - 525.46s | better than me with electronic thing it, uh, what's it's and thingamabobs. However, I do 528.32s - 534.32s | have a very particular set of skills. So, I'm going to use those skills to give myself a 534.32s - 541.29s | refund. Why? So, first we gotta get into these things, right? These are the dryers. That's the 545.95s - 551.13s | front view. That's the top view. Those are Phillips screws. And this is a Phillips 551.13s - 556.15s | screwdriver. This is a 20 minute talk. I'm not sure what you expected. The washers are a 558.61s - 562.59s | little bit harder, right? There's a tubular lock on the front of that thing and so we've got 562.59s - 568.09s | some options. The first option, hardcore mode. If you know how to pick tubular locks, that'll 568.09s - 572.69s | get you right in. But there's a problem with this. Now, there's an entire village of people 572.69s - 576.63s | here who would love to discuss the intricacies of lock pick along with you and then they 576.63s - 580.73s | would tell you not to pick any lock that doesn't belong to you. I'm not one of those 580.73s - 586.75s | people. Follow your heart. Do what the fuck you want to. Now, we can sidestep that issue 590.35s - 594.65s | though. Right? Inside the door of the washers there's a sticker that says Alliance Laundry 594.65s - 600.05s | Systems. Now I guess they supplied these machines to CSC before the retrofit. Uh and I might 600.05s - 604.53s | have got the idea to just type in Alliance Laundry Systems into Google. I'm not sure if you 604.53s - 609.83s | can read the fine print. Uh but that says Alliance Washer Dryer Key that I've typed into the 609.83s - 616.47s | search box and you only get one result and it's a key that costs $5.80 which I am more than 616.47s - 621.49s | willing to risk to to look cool up here on stage. And what


chunk_id: 215_chunk_12
chunk_text:
a problem with this. Now, there's an entire village of people 572.69s - 576.63s | here who would love to discuss the intricacies of lock pick along with you and then they 576.63s - 580.73s | would tell you not to pick any lock that doesn't belong to you. I'm not one of those 580.73s - 586.75s | people. Follow your heart. Do what the fuck you want to. Now, we can sidestep that issue 590.35s - 594.65s | though. Right? Inside the door of the washers there's a sticker that says Alliance Laundry 594.65s - 600.05s | Systems. Now I guess they supplied these machines to CSC before the retrofit. Uh and I might 600.05s - 604.53s | have got the idea to just type in Alliance Laundry Systems into Google. I'm not sure if you 604.53s - 609.83s | can read the fine print. Uh but that says Alliance Washer Dryer Key that I've typed into the 609.83s - 616.47s | search box and you only get one result and it's a key that costs $5.80 which I am more than 616.47s - 621.49s | willing to risk to to look cool up here on stage. And what do you know? Not only does it 625.49s - 631.75s | open this one, it opens them all. Now, once we're inside of course we have to actually give 631.75s - 638.69s | ourselves a refund somehow uh via subterfuge. And so there are some caveats here. Right? 638.69s - 643.69s | Chronologically this is where all the things that don't work would go in my story. Um and 643.69s - 647.57s | once you have physical access to a machine, hacking it really shouldn't be that hard. But 647.57s - 651.31s | there are some extra constraints here that make it difficult. First is that I can't break the 651.31s - 655.41s | machine. Right? My neighbors use these machines to do laundry. I like my neighbors and you 655.41s - 659.01s | really don't want to be on the bad side of someone who wakes up at 5 in the morning on a 659.01s - 664.15s | Sunday to do laundry. Really though, I can't break the machine because if I break the 664.15s - 667.19s | machine the property manager is going to get involved. They're going to call the board. The 667.19s - 671.39s | board


chunk_id: 215_chunk_13
chunk_text:
do you know? Not only does it 625.49s - 631.75s | open this one, it opens them all. Now, once we're inside of course we have to actually give 631.75s - 638.69s | ourselves a refund somehow uh via subterfuge. And so there are some caveats here. Right? 638.69s - 643.69s | Chronologically this is where all the things that don't work would go in my story. Um and 643.69s - 647.57s | once you have physical access to a machine, hacking it really shouldn't be that hard. But 647.57s - 651.31s | there are some extra constraints here that make it difficult. First is that I can't break the 651.31s - 655.41s | machine. Right? My neighbors use these machines to do laundry. I like my neighbors and you 655.41s - 659.01s | really don't want to be on the bad side of someone who wakes up at 5 in the morning on a 659.01s - 664.15s | Sunday to do laundry. Really though, I can't break the machine because if I break the 664.15s - 667.19s | machine the property manager is going to get involved. They're going to call the board. The 667.19s - 671.39s | board is made up of my neighbors. I like my neighbors. I don't want to waste their time. But 671.39s - 676.83s | seriously, I can't break the machines. If I break the machines the $2 joke doesn't go so 676.83s - 684.66s | well. What are they going to do? Sue me for $1,000? Sticker to the rescue. Inside the 684.66s - 688.64s | dryers we see uh they have another Alliance laundry system sticker. So all these things come 688.64s - 692.70s | from the same place. And you can't find too much information about the circuitry from 692.70s - 697.20s | Alliance's website. They don't have pictures of like the coin drop mechanism or anything. But 697.20s - 703.64s | Google does. Oh, sorry. There's only three wires coming out of this thing. How hard could it 703.64s - 708.58s | be? And we see inside the washers that we've got the same three wires going into a bullet 708.58s - 714.58s | connector. This is a paper clip with some tape around it. Just shove it in there. Ding ding 714.58s - 719.58s | ding. Each time you short it out you get credit for putting one


chunk_id: 215_chunk_14
chunk_text:
is made up of my neighbors. I like my neighbors. I don't want to waste their time. But 671.39s - 676.83s | seriously, I can't break the machines. If I break the machines the $2 joke doesn't go so 676.83s - 684.66s | well. What are they going to do? Sue me for $1,000? Sticker to the rescue. Inside the 684.66s - 688.64s | dryers we see uh they have another Alliance laundry system sticker. So all these things come 688.64s - 692.70s | from the same place. And you can't find too much information about the circuitry from 692.70s - 697.20s | Alliance's website. They don't have pictures of like the coin drop mechanism or anything. But 697.20s - 703.64s | Google does. Oh, sorry. There's only three wires coming out of this thing. How hard could it 703.64s - 708.58s | be? And we see inside the washers that we've got the same three wires going into a bullet 708.58s - 714.58s | connector. This is a paper clip with some tape around it. Just shove it in there. Ding ding 714.58s - 719.58s | ding. Each time you short it out you get credit for putting one quarter in. In seconds. It 720.18s - 726.43s | doesn't take weeks in snail mail to get a refund. Ha. The dryers are a little bit more 729.13s - 734.37s | difficult because uh you see there's no bullet connector here. But again uh when faced with 734.37s - 739.21s | adversity I find that it's always best to follow your heart and just do what the fuck you want 739.21s - 744.21s | to. And strip a little shielding off those fuckers. And just touch them together. It works. 744.73s - 749.73s | It's fine. Ding ding ding. And you get a refund from the dryers too. Now the next slide says the 754.61s - 759.55s | end but it's not really the end. Because of course there's a live demo that's pre-recorded 759.55s - 769.53s | uh but it's live. I just, it was live when I pre-recorded it. But it's live. What's up Defcon? 769.53s - 778.02s | We're doing full screen. This is important. And this is laundering money. I'm here in my 778.18s - 782.18s | building's laundry room at three in the morning on a Monday because I'm about to do absolutely 782.18s -


chunk_id: 215_chunk_15
chunk_text:
quarter in. In seconds. It 720.18s - 726.43s | doesn't take weeks in snail mail to get a refund. Ha. The dryers are a little bit more 729.13s - 734.37s | difficult because uh you see there's no bullet connector here. But again uh when faced with 734.37s - 739.21s | adversity I find that it's always best to follow your heart and just do what the fuck you want 739.21s - 744.21s | to. And strip a little shielding off those fuckers. And just touch them together. It works. 744.73s - 749.73s | It's fine. Ding ding ding. And you get a refund from the dryers too. Now the next slide says the 754.61s - 759.55s | end but it's not really the end. Because of course there's a live demo that's pre-recorded 759.55s - 769.53s | uh but it's live. I just, it was live when I pre-recorded it. But it's live. What's up Defcon? 769.53s - 778.02s | We're doing full screen. This is important. And this is laundering money. I'm here in my 778.18s - 782.18s | building's laundry room at three in the morning on a Monday because I'm about to do absolutely 782.18s - 787.18s | nothing wrong. Before we get started I've got to attach my phone to my Yopro so that you can 790.52s - 799.07s | see what I can see. So bear with me for a second. That should just about do it. And since I've 801.55s - 807.55s | already told you how this is gonna go uh we can just jump right into the demo. So over here on my 807.57s - 813.93s | right we see a bunch of stupid washing machines and the first thing that we need to do to get some 813.93s - 820.55s | free money out of these guys is to get into this service panel. Now obviously the best way to do 820.55s - 825.55s | that in terms of how cool it makes you look is with this Covert Instruments tubular lockpick. 827.95s - 833.35s | This is much more than a lockpick. It is actually an impressioning tool which means that you can 833.35s - 838.39s | save your game once you've picked the lock. And that is why I do not have to stand here for 30 838.39s - 845.03s | seconds and jiggle this thing back and


chunk_id: 215_chunk_16
chunk_text:
787.18s | nothing wrong. Before we get started I've got to attach my phone to my Yopro so that you can 790.52s - 799.07s | see what I can see. So bear with me for a second. That should just about do it. And since I've 801.55s - 807.55s | already told you how this is gonna go uh we can just jump right into the demo. So over here on my 807.57s - 813.93s | right we see a bunch of stupid washing machines and the first thing that we need to do to get some 813.93s - 820.55s | free money out of these guys is to get into this service panel. Now obviously the best way to do 820.55s - 825.55s | that in terms of how cool it makes you look is with this Covert Instruments tubular lockpick. 827.95s - 833.35s | This is much more than a lockpick. It is actually an impressioning tool which means that you can 833.35s - 838.39s | save your game once you've picked the lock. And that is why I do not have to stand here for 30 838.39s - 845.03s | seconds and jiggle this thing back and forth uh to get the lock open. Uh however you'll notice I've 845.03s - 849.77s | closed the panel back up and the reason for that is because as it turns out in the state of 849.77s - 854.77s | Maryland at least it is illegal to use a lockpick to commit a crime. And so instead I have here 856.57s - 861.59s | with me my great hundred key that I have purchased from Alliance Laundry Systems and what I'm 861.59s - 866.59s | going to do is open the service panel with this guy. Once it's open we can immediately identify 871.91s - 876.91s | the trio of red black and white wires running in from the coin drop mechanism and to perform the 878.77s - 884.25s | cha-ching operation I'm going to be using this custom hacking tool here that you can find at any 884.25s - 889.25s | Radio Shack. And all I'm going to do is insert the ends of it into the holes occupied by the 889.25s - 899.53s | red and black wires. Eventually the machine will go apeshit to indicate success and you can close 904.41s - 913.22s | up the service


chunk_id: 215_chunk_17
chunk_text:
forth uh to get the lock open. Uh however you'll notice I've 845.03s - 849.77s | closed the panel back up and the reason for that is because as it turns out in the state of 849.77s - 854.77s | Maryland at least it is illegal to use a lockpick to commit a crime. And so instead I have here 856.57s - 861.59s | with me my great hundred key that I have purchased from Alliance Laundry Systems and what I'm 861.59s - 866.59s | going to do is open the service panel with this guy. Once it's open we can immediately identify 871.91s - 876.91s | the trio of red black and white wires running in from the coin drop mechanism and to perform the 878.77s - 884.25s | cha-ching operation I'm going to be using this custom hacking tool here that you can find at any 884.25s - 889.25s | Radio Shack. And all I'm going to do is insert the ends of it into the holes occupied by the 889.25s - 899.53s | red and black wires. Eventually the machine will go apeshit to indicate success and you can close 904.41s - 913.22s | up the service panel once more. And that's it for the washers. Now we move on to the dryers. The 917.46s - 922.46s | dryers are protected by phillips screws and you will need a phillips screwdriver to remove them. 922.46s - 927.46s | If you do not have a phillips screwdriver I think you can find these on ebay but once we've got 933.67s - 938.67s | them removed the service panel comes right off and we once again identify the trio of red black and 943.01s - 948.01s | white wires. On this machine it looks like someone was kind enough to pre-strip the red and black 949.01s - 954.01s | wires and cover them with electrical tape. So all we have to do is remove this electrical tape and 960.45s - 965.45s | then to perform the cha-ching operation on the dryer I'm just going to smoosh the exposed portion 967.09s - 972.09s | together with my hands. And you'll notice I'm doing this with my bare hands and the reason for that is 973.05s - 978.05s | because it's actually safe to touch any exposed wires with your bare hands. Uh it's called grounding 980.53s - 1008.57s | look


chunk_id: 215_chunk_18
chunk_text:
panel once more. And that's it for the washers. Now we move on to the dryers. The 917.46s - 922.46s | dryers are protected by phillips screws and you will need a phillips screwdriver to remove them. 922.46s - 927.46s | If you do not have a phillips screwdriver I think you can find these on ebay but once we've got 933.67s - 938.67s | them removed the service panel comes right off and we once again identify the trio of red black and 943.01s - 948.01s | white wires. On this machine it looks like someone was kind enough to pre-strip the red and black 949.01s - 954.01s | wires and cover them with electrical tape. So all we have to do is remove this electrical tape and 960.45s - 965.45s | then to perform the cha-ching operation on the dryer I'm just going to smoosh the exposed portion 967.09s - 972.09s | together with my hands. And you'll notice I'm doing this with my bare hands and the reason for that is 973.05s - 978.05s | because it's actually safe to touch any exposed wires with your bare hands. Uh it's called grounding 980.53s - 1008.57s | look it up. Once again the loud screeching noise from the siren indicates that everything is fine. We 1011.61s - 1016.61s | can replace the electric tape and close the panel back up. Wait there's more. I'm so glad this 1016.73s - 1045.79s | happened. Defcon it's your boy back in the LR with an update. It is August 5th a few days before the 1058.75s - 1063.55s | conference and with all the publicity surrounding this talk it looks like CSC has upgraded the 1063.55s - 1067.39s | security on the machines in my building. Now it's not clear if this is a company wide 1067.39s - 1072.29s | initiative or just a personal fuck you to Michael Arlitsky and so to be on the safe side uh we're 1072.31s - 1077.31s | going to bypass it. The first thing that they've done is to replace the old shitty tubular locks on 1078.75s - 1084.15s | the washers with the new shitty tubular locks. So unfortunately the great hundred key will no 1084.15s - 1089.23s | longer open that up uh but if you have the covered instruments tubular lock pick of course uh it 1089.23s - 1094.23s | comes


chunk_id: 215_chunk_19
chunk_text:
it up. Once again the loud screeching noise from the siren indicates that everything is fine. We 1011.61s - 1016.61s | can replace the electric tape and close the panel back up. Wait there's more. I'm so glad this 1016.73s - 1045.79s | happened. Defcon it's your boy back in the LR with an update. It is August 5th a few days before the 1058.75s - 1063.55s | conference and with all the publicity surrounding this talk it looks like CSC has upgraded the 1063.55s - 1067.39s | security on the machines in my building. Now it's not clear if this is a company wide 1067.39s - 1072.29s | initiative or just a personal fuck you to Michael Arlitsky and so to be on the safe side uh we're 1072.31s - 1077.31s | going to bypass it. The first thing that they've done is to replace the old shitty tubular locks on 1078.75s - 1084.15s | the washers with the new shitty tubular locks. So unfortunately the great hundred key will no 1084.15s - 1089.23s | longer open that up uh but if you have the covered instruments tubular lock pick of course uh it 1089.23s - 1094.23s | comes right open. So fuck out of my face with that shit. On the dryers what they've done is to 1095.21s - 1100.21s | replace the Phillips screws with these security Torx screws. Now the nice thing about these is that 1103.55s - 1109.03s | if you don't have the bit you can buy a set on Amazon for about $6.99 and the nice thing about 1109.03s - 1114.29s | $6.99 is that if you don't have it what you can do is completely ignore everything that I just 1114.29s - 1119.97s | said and get a standard flat head screwdriver and just shove it in that bitch. Now what I want you 1119.97s - 1124.71s | to do next is close your eyes and acknowledge that through the power of Christ anything is 1124.71s - 1141.88s | possible and then slowly begin to turn the screwdriver. Inside the only changes that have been 1141.88s - 1146.86s | made are that someone has gone just fucking bonkers on this thing with a roll of electrical 1146.86s - 1151.86s | tape but that should not stop anyone who is not an idiot. Happy DEF CON. This is all completely 1164.26s - 1165.16s


chunk_id: 215_chunk_20
chunk_text:
right open. So fuck out of my face with that shit. On the dryers what they've done is to 1095.21s - 1100.21s | replace the Phillips screws with these security Torx screws. Now the nice thing about these is that 1103.55s - 1109.03s | if you don't have the bit you can buy a set on Amazon for about $6.99 and the nice thing about 1109.03s - 1114.29s | $6.99 is that if you don't have it what you can do is completely ignore everything that I just 1114.29s - 1119.97s | said and get a standard flat head screwdriver and just shove it in that bitch. Now what I want you 1119.97s - 1124.71s | to do next is close your eyes and acknowledge that through the power of Christ anything is 1124.71s - 1141.88s | possible and then slowly begin to turn the screwdriver. Inside the only changes that have been 1141.88s - 1146.86s | made are that someone has gone just fucking bonkers on this thing with a roll of electrical 1146.86s - 1151.86s | tape but that should not stop anyone who is not an idiot. Happy DEF CON. This is all completely 1164.26s - 1165.16s | legal thank you.


chunk_id: 216_chunk_0
chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=5S_1uHLn4SI", "title": "DEF CON 32 - Finding & exploiting local attacks on 1Password Mac desktop app - J. Hoffman, C. Morgan", "description": "Password managers are routinely granted a massive level of trust from users, by nature of managing some of their most sensitive credentials. For any noteworthy password manager, the encryption standards for user data are well understood and highly scrutinized. What is less understood is the attack surface of the software itself. This presentation explores the local security of the 1Password MacOS desktop application and answers the question of \u201chow safe are my passwords if my computer is infected or otherwise compromised?\u201d.\n\nThis talk will cover the outcome of our research into 1Password, presenting several different attacks to dump local 1Password vaults. This includes describing multiple application vulnerabilities and security weaknesses we identified in the 1Password MacOS desktop application, as well as discussing the inherent limitations in its usage of IPC mechanisms and open source software. Additionally, we will discuss novel vulnerabilities found in Google Chrome that aided our exploitation of the 1Password browser extension.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 2330, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 3.25s - 8.25s | This text was transcribed using whisper model: large-v2 Hello. Hi everyone. Uh, I'm Colby Morgan. I'm here with Jeffrey Hoffman. We're both 9.59s - 14.77s | offensive security engineers at Robinhood. And earlier this year we did some research into 14.77s - 20.67s | 1Password for Mac. Uh, with the goal of identifying local attacks to dump vaults. And 20.67s - 23.91s | today we're gonna be sharing with you the outcome of our research and some of the uh, 23.91s - 31.03s | vulnerabilities we discovered. So, why look at 1Password for Mac? Um, well if you use 31.07s - 35.17s | 1Password, it's probably a place where you have some of your most sensitive credentials. 35.17s - 38.71s | And we know that APTs, red teamers, one of the first things they're gonna do when they 38.71s - 44.21s | land on an endpoint is look for credentials. Things like uh, cloud credentials, browser 44.21s - 48.25s | cookies, um, passwords, API keys, things they might be able to use even if they were 48.25s - 54.39s | evicted from the laptop uh, to retain um, some sort of access or even more access. Um, if 54.39s - 58.79s | they were to be grepping around


chunk_id: 216_chunk_1
chunk_text:
Hello. Hi everyone. Uh, I'm Colby Morgan. I'm here with Jeffrey Hoffman. We're both 9.59s - 14.77s | offensive security engineers at Robinhood. And earlier this year we did some research into 14.77s - 20.67s | 1Password for Mac. Uh, with the goal of identifying local attacks to dump vaults. And 20.67s - 23.91s | today we're gonna be sharing with you the outcome of our research and some of the uh, 23.91s - 31.03s | vulnerabilities we discovered. So, why look at 1Password for Mac? Um, well if you use 31.07s - 35.17s | 1Password, it's probably a place where you have some of your most sensitive credentials. 35.17s - 38.71s | And we know that APTs, red teamers, one of the first things they're gonna do when they 38.71s - 44.21s | land on an endpoint is look for credentials. Things like uh, cloud credentials, browser 44.21s - 48.25s | cookies, um, passwords, API keys, things they might be able to use even if they were 48.25s - 54.39s | evicted from the laptop uh, to retain um, some sort of access or even more access. Um, if 54.39s - 58.79s | they were to be grepping around uh, looking for files that have the word password in the 58.79s - 64.29s | file name, one thing they would find is this 1Password dot SQLite file. And this is a 64.29s - 69.69s | SQLite database uh, it's encrypted but it contains your uh, 1Password vault contents. So 69.69s - 74.53s | that file itself wouldn't get you much. Um, so we're gonna talk about some attacks later 74.53s - 79.97s | that might be able to get you some of that content. Um, so to talk a little bit about the 79.97s - 84.55s | local security model of 1Password on Mac. Um, when you log in with your password to 84.55s - 89.21s | 1Password, uh, it doesn't actually store your password. It's not even sending your password to 89.21s - 95.35s | the server. Instead what's going on is um, your password along with a locally stored secret 95.35s - 99.93s | key are combined cryptographically to generate a master unlock key uh, which is now 99.93s - 103.97s | referred to as an account unlock key. You'll hear us throughout this talk refer to it as a 103.97s - 110.97s | muck because that's


chunk_id: 216_chunk_2
chunk_text:
uh, looking for files that have the word password in the 58.79s - 64.29s | file name, one thing they would find is this 1Password dot SQLite file. And this is a 64.29s - 69.69s | SQLite database uh, it's encrypted but it contains your uh, 1Password vault contents. So 69.69s - 74.53s | that file itself wouldn't get you much. Um, so we're gonna talk about some attacks later 74.53s - 79.97s | that might be able to get you some of that content. Um, so to talk a little bit about the 79.97s - 84.55s | local security model of 1Password on Mac. Um, when you log in with your password to 84.55s - 89.21s | 1Password, uh, it doesn't actually store your password. It's not even sending your password to 89.21s - 95.35s | the server. Instead what's going on is um, your password along with a locally stored secret 95.35s - 99.93s | key are combined cryptographically to generate a master unlock key uh, which is now 99.93s - 103.97s | referred to as an account unlock key. You'll hear us throughout this talk refer to it as a 103.97s - 110.97s | muck because that's how we saw it referenced in the code. Um, so we use SSO meaning uh, we 110.99s - 116.83s | don't have a password for 1Password and how it works in this case is uh, you authenticate and 116.83s - 122.67s | then you get a device key um, and you receive a credential bundle from 1Password's API and 122.67s - 127.17s | the device key is stored in your Mac keychain and that is used to decrypt the credential 127.17s - 132.65s | bundle that contains that muck that I referenced. So the muck is basically the keys to the 132.65s - 138.65s | kingdom. It allows you to decrypt that SQLite database where your vault content is stored. 138.67s - 144.11s | Malware on your machine uh, without a SIP bypass on Mac or a way to inject into processes 144.11s - 149.31s | arbitrarily shouldn't be able to get your vault contents uh, decrypted um, without in the 149.31s - 154.85s | SSO case unrestricted keychain access and in the non-SSO case um, access to your 1Password 154.85s - 160.59s | password meaning something you know um, and your locally stored secret key. Once you have 160.59s


chunk_id: 216_chunk_3
chunk_text:
how we saw it referenced in the code. Um, so we use SSO meaning uh, we 110.99s - 116.83s | don't have a password for 1Password and how it works in this case is uh, you authenticate and 116.83s - 122.67s | then you get a device key um, and you receive a credential bundle from 1Password's API and 122.67s - 127.17s | the device key is stored in your Mac keychain and that is used to decrypt the credential 127.17s - 132.65s | bundle that contains that muck that I referenced. So the muck is basically the keys to the 132.65s - 138.65s | kingdom. It allows you to decrypt that SQLite database where your vault content is stored. 138.67s - 144.11s | Malware on your machine uh, without a SIP bypass on Mac or a way to inject into processes 144.11s - 149.31s | arbitrarily shouldn't be able to get your vault contents uh, decrypted um, without in the 149.31s - 154.85s | SSO case unrestricted keychain access and in the non-SSO case um, access to your 1Password 154.85s - 160.59s | password meaning something you know um, and your locally stored secret key. Once you have 160.59s - 165.33s | the muck uh, the full decryption process um, isn't super straightforward but it's fully 165.35s - 171.05s | explained in this blog post from David Shoots aka Darthnal uh, where he also had some code 171.05s - 176.90s | that we referenced for the decryption and so this was really helpful in our research. 176.90s - 180.94s | Yeah, so it's generally understood that if somebody has full control over your machine 180.94s - 185.04s | they're gonna be able to access whatever data it protects in some way shape or form. That 185.04s - 189.08s | being said, 1Password still cares about creating meaningful security boundaries wherever 189.08s - 194.82s | possible and on Mac OS specifically with things like SIP and code signing uh, there is a 194.84s - 199.20s | possibility to create these significant boundaries. Uh, these are just some CVEs that are Mac 199.20s - 203.74s | specific uh, we took a look at these CVEs when we first started our work and they indicated 203.74s - 208.52s | to us that we were thinking about the right things and looking in the right places. So when 208.52s - 212.18s | we


chunk_id: 216_chunk_4
chunk_text:
- 165.33s | the muck uh, the full decryption process um, isn't super straightforward but it's fully 165.35s - 171.05s | explained in this blog post from David Shoots aka Darthnal uh, where he also had some code 171.05s - 176.90s | that we referenced for the decryption and so this was really helpful in our research. 176.90s - 180.94s | Yeah, so it's generally understood that if somebody has full control over your machine 180.94s - 185.04s | they're gonna be able to access whatever data it protects in some way shape or form. That 185.04s - 189.08s | being said, 1Password still cares about creating meaningful security boundaries wherever 189.08s - 194.82s | possible and on Mac OS specifically with things like SIP and code signing uh, there is a 194.84s - 199.20s | possibility to create these significant boundaries. Uh, these are just some CVEs that are Mac 199.20s - 203.74s | specific uh, we took a look at these CVEs when we first started our work and they indicated 203.74s - 208.52s | to us that we were thinking about the right things and looking in the right places. So when 208.52s - 212.18s | we started out looking at 1Password we didn't really look at the app, we looked at the 212.18s - 217.92s | framework that it was built on. Uh, we looked at the framework that it was built on uh, which 217.92s - 222.92s | is Electron and so Electron is a chromium and node based framework. It has a pretty well 222.92s - 227.90s | defined attack surface, the standard attack you'll see is somebody trying to get XSS and 227.90s - 232.64s | convert that into RCE or something else but 1Password is an app, doesn't have a whole lot of 232.64s - 238.50s | opportunities to inject user controlled content. Uh, the XSS to RCE route is usually 238.50s - 243.22s | considered uh, a remote attack but because we're local attackers we have the ability to 243.22s - 247.72s | consider local things and one of the local security controls in Electron is something 247.72s - 253.00s | called fuses. Uh, these fuses control the behavior of 1Password in certain situations. I'm 253.00s - 256.96s | not going to read all of these off to you but I'll tell you that 1Password correctly set all 256.96s - 261.80s


chunk_id: 216_chunk_5
chunk_text:
started out looking at 1Password we didn't really look at the app, we looked at the 212.18s - 217.92s | framework that it was built on. Uh, we looked at the framework that it was built on uh, which 217.92s - 222.92s | is Electron and so Electron is a chromium and node based framework. It has a pretty well 222.92s - 227.90s | defined attack surface, the standard attack you'll see is somebody trying to get XSS and 227.90s - 232.64s | convert that into RCE or something else but 1Password is an app, doesn't have a whole lot of 232.64s - 238.50s | opportunities to inject user controlled content. Uh, the XSS to RCE route is usually 238.50s - 243.22s | considered uh, a remote attack but because we're local attackers we have the ability to 243.22s - 247.72s | consider local things and one of the local security controls in Electron is something 247.72s - 253.00s | called fuses. Uh, these fuses control the behavior of 1Password in certain situations. I'm 253.00s - 256.96s | not going to read all of these off to you but I'll tell you that 1Password correctly set all 256.96s - 261.80s | of these. Uh, but some of the interesting ones that, that you might take a look at are things 261.80s - 267.64s | like only load app from ASAR. Uh, if this had been enabled or rather disabled, uh, you might 267.64s - 272.78s | be able to proxy the app and inject JavaScript that way. And the enable node CLI inspect 272.78s - 277.76s | arguments fuse is also pretty interesting. It controls whether or not this Chromium flag 277.76s - 283.12s | called the remote debugging port works. If that had been enabled in 1Password, we would 283.12s - 288.16s | have, uh, started it with this flag, uh, connected a Chromium debugger to it and from there 288.16s - 292.34s | you can see all the pages that the app has. You would have been able to inject, uh, 292.34s - 298.54s | JavaScript or change the currently running JavaScript in any of those pages. So, once we 298.54s - 303.24s | saw that all the electron, uh, attack surface was, was considered and wasn't going to be 303.24s - 308.24s | viable, we started looking at how the app functioned and there were 3 main


chunk_id: 216_chunk_6
chunk_text:
| of these. Uh, but some of the interesting ones that, that you might take a look at are things 261.80s - 267.64s | like only load app from ASAR. Uh, if this had been enabled or rather disabled, uh, you might 267.64s - 272.78s | be able to proxy the app and inject JavaScript that way. And the enable node CLI inspect 272.78s - 277.76s | arguments fuse is also pretty interesting. It controls whether or not this Chromium flag 277.76s - 283.12s | called the remote debugging port works. If that had been enabled in 1Password, we would 283.12s - 288.16s | have, uh, started it with this flag, uh, connected a Chromium debugger to it and from there 288.16s - 292.34s | you can see all the pages that the app has. You would have been able to inject, uh, 292.34s - 298.54s | JavaScript or change the currently running JavaScript in any of those pages. So, once we 298.54s - 303.24s | saw that all the electron, uh, attack surface was, was considered and wasn't going to be 303.24s - 308.24s | viable, we started looking at how the app functioned and there were 3 main ways that we saw 308.24s - 314.98s | to get passwords out of the desktop app and those were the SSH agent, the CLI, and the 314.98s - 320.12s | browser extension. Now the SSH agent and the CLI were disabled by default, but the browser 320.12s - 325.64s | extension is enabled by default. And so once we saw this, we installed the browser extension 325.64s - 329.44s | and as long as the app is unlocked, if you click the browser extension, it's going to 329.44s - 332.60s | seamlessly sync all of your passwords and they're going to show up without any kind of 332.60s - 336.34s | re-authentication. And so as soon as we saw that behavior, we started trying to figure out 336.34s - 342.24s | how we could abuse that. And so we briefly considered the idea of maybe if we had some 342.24s - 347.08s | cool RCE and sandbox escape, we could control Chrome and try to do that remotely. Those are 347.08s - 351.56s | prohibitively expensive and again, because we're local attackers, we have much less 351.58s - 355.88s | expensive options. And uh, that fuse that I mentioned before


chunk_id: 216_chunk_7
chunk_text:
ways that we saw 308.24s - 314.98s | to get passwords out of the desktop app and those were the SSH agent, the CLI, and the 314.98s - 320.12s | browser extension. Now the SSH agent and the CLI were disabled by default, but the browser 320.12s - 325.64s | extension is enabled by default. And so once we saw this, we installed the browser extension 325.64s - 329.44s | and as long as the app is unlocked, if you click the browser extension, it's going to 329.44s - 332.60s | seamlessly sync all of your passwords and they're going to show up without any kind of 332.60s - 336.34s | re-authentication. And so as soon as we saw that behavior, we started trying to figure out 336.34s - 342.24s | how we could abuse that. And so we briefly considered the idea of maybe if we had some 342.24s - 347.08s | cool RCE and sandbox escape, we could control Chrome and try to do that remotely. Those are 347.08s - 351.56s | prohibitively expensive and again, because we're local attackers, we have much less 351.58s - 355.88s | expensive options. And uh, that fuse that I mentioned before that controlled the remote 355.88s - 361.32s | debugging port, that's a Chromium flag. And so while Electron can turn that off, Chrome and 361.32s - 366.50s | other Chromium based browsers can't. It's just how they work. So as soon as we realized 366.50s - 371.40s | that, we found the JavaScript responsible for doing the syncing. It's that green URL that's 371.40s - 376.40s | on screen now. And we opened it up, looked at the console and we saw some log messages 376.40s - 381.14s | related to native core. This is a native messaging host and we'll re- we'll revisit that in a 381.16s - 387.76s | little bit. Uh, but essentially what we did is we read the JavaScript from that page, uh, 387.76s - 392.80s | beautified it, modified it to do that syncing behavior immediately and with the remote 392.80s - 397.74s | debugging port, we just injected our new extension into that page and uh, everything was 397.74s - 404.04s | synced and we were able to extract passwords and decrypt them. And so this is our mental 404.04s - 408.62s | model of 1Password. We'll revisit this image a couple of


chunk_id: 216_chunk_8
chunk_text:
that controlled the remote 355.88s - 361.32s | debugging port, that's a Chromium flag. And so while Electron can turn that off, Chrome and 361.32s - 366.50s | other Chromium based browsers can't. It's just how they work. So as soon as we realized 366.50s - 371.40s | that, we found the JavaScript responsible for doing the syncing. It's that green URL that's 371.40s - 376.40s | on screen now. And we opened it up, looked at the console and we saw some log messages 376.40s - 381.14s | related to native core. This is a native messaging host and we'll re- we'll revisit that in a 381.16s - 387.76s | little bit. Uh, but essentially what we did is we read the JavaScript from that page, uh, 387.76s - 392.80s | beautified it, modified it to do that syncing behavior immediately and with the remote 392.80s - 397.74s | debugging port, we just injected our new extension into that page and uh, everything was 397.74s - 404.04s | synced and we were able to extract passwords and decrypt them. And so this is our mental 404.04s - 408.62s | model of 1Password. We'll revisit this image a couple of times during the presentation. Uh, 408.66s - 413.30s | but what's important to note here is that we didn't really have a vulnerability in 1Password 413.30s - 422.15s | exactly. We're kind of just abusing how Chrome and extensions function. Yeah, so sticking 422.15s - 426.53s | along the path of attacking the browser extension, um, one question we were asking 426.53s - 430.27s | ourselves is would we be able to spoof the legitimate 1Password extension with our own 430.27s - 436.23s | extension or make changes to the 1Password extension and still uh, still have it do that 436.27s - 442.27s | automatic syncing process Jeff, uh, referred to before. Um, so if you're not familiar, uh, you 442.27s - 447.01s | can start Chrome from the command line, uh, along with this dash dash load extension flag to 447.01s - 451.75s | load extensions from disk, um, which are referred to as unpacked extensions. So what we 451.75s - 456.59s | were wondering is if we could load a modified version of 1Password from disk, uh, and have it 456.59s - 460.79s | do that syncing process, obviously the modified code we would add would be JavaScript


chunk_id: 216_chunk_9
chunk_text:
times during the presentation. Uh, 408.66s - 413.30s | but what's important to note here is that we didn't really have a vulnerability in 1Password 413.30s - 422.15s | exactly. We're kind of just abusing how Chrome and extensions function. Yeah, so sticking 422.15s - 426.53s | along the path of attacking the browser extension, um, one question we were asking 426.53s - 430.27s | ourselves is would we be able to spoof the legitimate 1Password extension with our own 430.27s - 436.23s | extension or make changes to the 1Password extension and still uh, still have it do that 436.27s - 442.27s | automatic syncing process Jeff, uh, referred to before. Um, so if you're not familiar, uh, you 442.27s - 447.01s | can start Chrome from the command line, uh, along with this dash dash load extension flag to 447.01s - 451.75s | load extensions from disk, um, which are referred to as unpacked extensions. So what we 451.75s - 456.59s | were wondering is if we could load a modified version of 1Password from disk, uh, and have it 456.59s - 460.79s | do that syncing process, obviously the modified code we would add would be JavaScript that 460.79s - 464.27s | makes it do what we want, such as like taking those passwords after they sync and sending 464.27s - 469.11s | them to a remote server. Um, there were a couple of kind of challenges that, uh, made this 469.11s - 475.41s | not straightforward. So, first, uh, the, when the extension talks to the Mac OS 1Password 475.41s - 479.05s | app, uh, the 1Password app is going to verify that the connection is coming from the 479.05s - 484.49s | legitimate extension's extension identifier. Um, and then further, we have a managed Chrome 484.49s - 489.09s | policy that prevents, uh, loading unpacked extensions altogether, meaning we couldn't 489.09s - 494.79s | launch Chrome with that CLI flag. Um, and also, all extensions are disallowed by default, with 494.79s - 500.77s | only some being allow listed. So we started trying to tackle these challenges one by one. Uh, 500.77s - 506.03s | first, uh, spoofing the extension identifier. Thankfully this was extremely straightforward 506.03s - 511.31s | because Google tells us exactly how to do it on their browser extension, uh, security FAQ 511.31s - 517.59s | page, where they basically explain


chunk_id: 216_chunk_10
chunk_text:
that 460.79s - 464.27s | makes it do what we want, such as like taking those passwords after they sync and sending 464.27s - 469.11s | them to a remote server. Um, there were a couple of kind of challenges that, uh, made this 469.11s - 475.41s | not straightforward. So, first, uh, the, when the extension talks to the Mac OS 1Password 475.41s - 479.05s | app, uh, the 1Password app is going to verify that the connection is coming from the 479.05s - 484.49s | legitimate extension's extension identifier. Um, and then further, we have a managed Chrome 484.49s - 489.09s | policy that prevents, uh, loading unpacked extensions altogether, meaning we couldn't 489.09s - 494.79s | launch Chrome with that CLI flag. Um, and also, all extensions are disallowed by default, with 494.79s - 500.77s | only some being allow listed. So we started trying to tackle these challenges one by one. Uh, 500.77s - 506.03s | first, uh, spoofing the extension identifier. Thankfully this was extremely straightforward 506.03s - 511.31s | because Google tells us exactly how to do it on their browser extension, uh, security FAQ 511.31s - 517.59s | page, where they basically explain that they don't consider this a bug. And, um, basically, 517.61s - 523.19s | um, every browser extension has a manifest.json file with a bunch of key entries. Um, one of 523.19s - 528.93s | those is called key, and, um, basically, if you take that key entry and put it in your own 528.93s - 533.69s | extension's manifest.json file, the extension IDs will be the same because the extension 533.69s - 539.07s | identifier is derived from that key entry. So we can just grab the legitimate extension's, um, 539.07s - 544.01s | key identifier, put that on our own extension, and then it'll match. So now we're left with 544.03s - 548.53s | the challenges from the managed Chrome policy. Uh, we basically started just trying to 548.53s - 554.11s | figure out how it worked and how it is enforced, and we came across this policy directory in 554.11s - 559.37s | Google Chrome's, um, application support directory. And we decided to just remove it and see 559.37s - 565.51s | what happens. Um, so, if you delete that policy directory and you start Chrome, Chrome is 565.51s - 569.29s | actually smart enough to realize


chunk_id: 216_chunk_11
chunk_text:
that they don't consider this a bug. And, um, basically, 517.61s - 523.19s | um, every browser extension has a manifest.json file with a bunch of key entries. Um, one of 523.19s - 528.93s | those is called key, and, um, basically, if you take that key entry and put it in your own 528.93s - 533.69s | extension's manifest.json file, the extension IDs will be the same because the extension 533.69s - 539.07s | identifier is derived from that key entry. So we can just grab the legitimate extension's, um, 539.07s - 544.01s | key identifier, put that on our own extension, and then it'll match. So now we're left with 544.03s - 548.53s | the challenges from the managed Chrome policy. Uh, we basically started just trying to 548.53s - 554.11s | figure out how it worked and how it is enforced, and we came across this policy directory in 554.11s - 559.37s | Google Chrome's, um, application support directory. And we decided to just remove it and see 559.37s - 565.51s | what happens. Um, so, if you delete that policy directory and you start Chrome, Chrome is 565.51s - 569.29s | actually smart enough to realize it should be there and it'll recreate it and it'll start 569.29s - 574.47s | reinforcing itself. However, um, as I mentioned before, we were prevented from starting 574.47s - 578.67s | Chrome with the dash dash load extension flag, but what you can do is delete the policy 578.67s - 583.05s | directory, start Chrome with the dash dash load extension flag, which will load your extension, 583.05s - 588.25s | um, and it, that works because the policy directory doesn't exist at the time Chrome starts, 588.25s - 592.15s | but then Chrome will recreate that directory and start reinforcing itself, um, but your 592.15s - 596.33s | extension is already loaded, um, but Chrome will start going through and saying which 596.33s - 601.13s | extension should be allowed, if it's not allowed, remove it. Um, but thankfully, it's only 601.13s - 605.91s | doing a shallow check of comparing the extension identifier to the allow list, and we're 605.91s - 610.41s | spoofing the legitimate extension's extension identifier, so it doesn't remove our modified 610.41s - 617.41s | 1Password. Um, so that overcomes all of the challenges we had, and here is us just, uh, 617.41s


chunk_id: 216_chunk_12
chunk_text:
it should be there and it'll recreate it and it'll start 569.29s - 574.47s | reinforcing itself. However, um, as I mentioned before, we were prevented from starting 574.47s - 578.67s | Chrome with the dash dash load extension flag, but what you can do is delete the policy 578.67s - 583.05s | directory, start Chrome with the dash dash load extension flag, which will load your extension, 583.05s - 588.25s | um, and it, that works because the policy directory doesn't exist at the time Chrome starts, 588.25s - 592.15s | but then Chrome will recreate that directory and start reinforcing itself, um, but your 592.15s - 596.33s | extension is already loaded, um, but Chrome will start going through and saying which 596.33s - 601.13s | extension should be allowed, if it's not allowed, remove it. Um, but thankfully, it's only 601.13s - 605.91s | doing a shallow check of comparing the extension identifier to the allow list, and we're 605.91s - 610.41s | spoofing the legitimate extension's extension identifier, so it doesn't remove our modified 610.41s - 617.41s | 1Password. Um, so that overcomes all of the challenges we had, and here is us just, uh, 617.41s - 622.95s | adding a POC.html file to the extension. We have the legitimate extension identifier, so 622.97s - 627.45s | our passwords do sync when 1Password is unlocked, um, and it pulls in all our passwords as 627.45s - 631.95s | expected, and obviously, we could have also added, um, edited the JavaScript or added 631.95s - 638.40s | JavaScript to, uh, make it grab those credentials and send them elsewhere. 638.40s - 643.04s | So earlier, I brought up the term native messaging host, and what that is is a binary that 643.04s - 647.70s | acts as a middleman between, uh, Chrome or other browsers that support this protocol and 647.70s - 653.92s | the desktop app, and, uh, how it normally works is Chrome will launch it, it will, uh, hold 653.92s - 658.08s | pipes to the standard in and standard out of that process, and then send JSON messages back 658.08s - 664.28s | and forth. And if you're familiar with IPC on macOS specifically, that's a big deviation 664.28s - 669.02s | from the norm. Normally, things use XPC, and part of the reason they do that is because when 669.02s - 673.60s


chunk_id: 216_chunk_13
chunk_text:
- 622.95s | adding a POC.html file to the extension. We have the legitimate extension identifier, so 622.97s - 627.45s | our passwords do sync when 1Password is unlocked, um, and it pulls in all our passwords as 627.45s - 631.95s | expected, and obviously, we could have also added, um, edited the JavaScript or added 631.95s - 638.40s | JavaScript to, uh, make it grab those credentials and send them elsewhere. 638.40s - 643.04s | So earlier, I brought up the term native messaging host, and what that is is a binary that 643.04s - 647.70s | acts as a middleman between, uh, Chrome or other browsers that support this protocol and 647.70s - 653.92s | the desktop app, and, uh, how it normally works is Chrome will launch it, it will, uh, hold 653.92s - 658.08s | pipes to the standard in and standard out of that process, and then send JSON messages back 658.08s - 664.28s | and forth. And if you're familiar with IPC on macOS specifically, that's a big deviation 664.28s - 669.02s | from the norm. Normally, things use XPC, and part of the reason they do that is because when 669.02s - 673.60s | you use XPC, you have something called an audit token available to you, and you can use 673.60s - 677.54s | that audit token to securely look up the code signing information of the process sending 677.54s - 682.34s | the message. Because there's no audit token, and there's no way to attach it to this method 682.36s - 688.84s | of IPC, uh, there's not really a great way for 1Password to go and verify the code signing 688.84s - 693.04s | information of whatever is sending that JSON. And they seem to know about this and, and 693.04s - 698.48s | still try to implement, uh, a parent PID check, and essentially what they did was they would 698.48s - 702.78s | call getPPID, go look up the code signing information of the parent, and check it a bunch 702.78s - 708.82s | of, uh, check it across the list of accepted and trusted browsers. But that's a generally 708.84s - 714.18s | understood, you know, anti-pattern on XPC. The reason they tried it here is, again, there's 714.18s - 720.82s | no audit token, so they can't really do much better. Uh, so, going forward, we're


chunk_id: 216_chunk_14
chunk_text:
| you use XPC, you have something called an audit token available to you, and you can use 673.60s - 677.54s | that audit token to securely look up the code signing information of the process sending 677.54s - 682.34s | the message. Because there's no audit token, and there's no way to attach it to this method 682.36s - 688.84s | of IPC, uh, there's not really a great way for 1Password to go and verify the code signing 688.84s - 693.04s | information of whatever is sending that JSON. And they seem to know about this and, and 693.04s - 698.48s | still try to implement, uh, a parent PID check, and essentially what they did was they would 698.48s - 702.78s | call getPPID, go look up the code signing information of the parent, and check it a bunch 702.78s - 708.82s | of, uh, check it across the list of accepted and trusted browsers. But that's a generally 708.84s - 714.18s | understood, you know, anti-pattern on XPC. The reason they tried it here is, again, there's 714.18s - 720.82s | no audit token, so they can't really do much better. Uh, so, going forward, we're going to 720.82s - 724.42s | talk about dumping, and instrumenting, and hooking, and our tool of choice for that was 724.42s - 728.26s | Frida. We're going to release some Frida scripts for people to look at other native 728.26s - 735.30s | messaging hosts. Um, what you can see on screen here is me hooking the read and write 735.32s - 739.76s | methods for the native messaging host, and this is how we figured out what JSON to send 739.76s - 745.36s | when we actually wrote our exploit. And so, like I said, it's a well understood problem 745.36s - 750.04s | that if you're doing XPC, you should be using the audit token because if you try to look up 750.04s - 754.90s | the code signing information by PID, what can happen is someone will send a message, exec 754.90s - 760.54s | into the actual process that was supposed to send that message, and by the time the PID is 760.56s - 765.76s | used, the co-signing information will be that of the trusted process and not the malicious 765.76s - 770.26s | one that sent that initial message. And so we considered


chunk_id: 216_chunk_15
chunk_text:
going to 720.82s - 724.42s | talk about dumping, and instrumenting, and hooking, and our tool of choice for that was 724.42s - 728.26s | Frida. We're going to release some Frida scripts for people to look at other native 728.26s - 735.30s | messaging hosts. Um, what you can see on screen here is me hooking the read and write 735.32s - 739.76s | methods for the native messaging host, and this is how we figured out what JSON to send 739.76s - 745.36s | when we actually wrote our exploit. And so, like I said, it's a well understood problem 745.36s - 750.04s | that if you're doing XPC, you should be using the audit token because if you try to look up 750.04s - 754.90s | the code signing information by PID, what can happen is someone will send a message, exec 754.90s - 760.54s | into the actual process that was supposed to send that message, and by the time the PID is 760.56s - 765.76s | used, the co-signing information will be that of the trusted process and not the malicious 765.76s - 770.26s | one that sent that initial message. And so we considered doing that, but we actually don't 770.26s - 775.04s | have to, there's a, a bunch better way. What you can do is create this process tree of 775.04s - 780.04s | three processes, where the parent holds the standard in and standard out write pipes, the 780.04s - 785.24s | child execs into a trusted browser, and the grandchild is actually the browser support 785.28s - 792.02s | process. And when it goes to do that, uh, get PPID check, we've satisfied that condition. So 792.02s - 796.16s | this is what the exploit looks like in process, and once you've set up these pipes, you can 796.16s - 802.66s | just send the JSON that we showed, uh, in the Frida slide, and the browser support process 802.66s - 809.00s | will happily return back to you the muck. And so here's our mental model again. We've 809.00s - 812.88s | totally cut Chrome and the need for the actual extension out of the equation, this time 812.88s - 817.16s | we're attacking browser support. And we were satisfied with this, this is definitely a 817.16s - 821.96s | valid way to get the muck and, and dump


chunk_id: 216_chunk_16
chunk_text:
doing that, but we actually don't 770.26s - 775.04s | have to, there's a, a bunch better way. What you can do is create this process tree of 775.04s - 780.04s | three processes, where the parent holds the standard in and standard out write pipes, the 780.04s - 785.24s | child execs into a trusted browser, and the grandchild is actually the browser support 785.28s - 792.02s | process. And when it goes to do that, uh, get PPID check, we've satisfied that condition. So 792.02s - 796.16s | this is what the exploit looks like in process, and once you've set up these pipes, you can 796.16s - 802.66s | just send the JSON that we showed, uh, in the Frida slide, and the browser support process 802.66s - 809.00s | will happily return back to you the muck. And so here's our mental model again. We've 809.00s - 812.88s | totally cut Chrome and the need for the actual extension out of the equation, this time 812.88s - 817.16s | we're attacking browser support. And we were satisfied with this, this is definitely a 817.16s - 821.96s | valid way to get the muck and, and dump all your vaults, but we thought, well what if we 821.96s - 825.80s | could also cut out browser support? So we started looking at the browser helper XPC 825.80s - 832.78s | server. And I've talked a lot about XPC already, I'll spare you, but they do everything 832.78s - 836.52s | correctly here that you would expect. They take the audit token, they get the co-signing 836.52s - 841.60s | information in a secure way, and they check the team identifier and bundle ID. 841.60s - 846.54s | Unfortunately, what they forgot to do was consider, uh, the hardened runtime, or the 846.54s - 850.58s | co-signing flags of the client. And so, if you're not familiar with hardened runtime, 850.58s - 855.58s | essentially what it is, is a co-signing flag that says to macOS to, uh, block all different 857.72s - 863.62s | manners of injecting, uh, code dynamically, and that makes sense, right? If your security 863.62s - 869.16s | model is going to be built around co-signing, you don't want ways to have untrusted code 869.16s - 874.10s | running in a process without invalidating that signature. And so that was introduced in 874.10s


chunk_id: 216_chunk_17
chunk_text:
all your vaults, but we thought, well what if we 821.96s - 825.80s | could also cut out browser support? So we started looking at the browser helper XPC 825.80s - 832.78s | server. And I've talked a lot about XPC already, I'll spare you, but they do everything 832.78s - 836.52s | correctly here that you would expect. They take the audit token, they get the co-signing 836.52s - 841.60s | information in a secure way, and they check the team identifier and bundle ID. 841.60s - 846.54s | Unfortunately, what they forgot to do was consider, uh, the hardened runtime, or the 846.54s - 850.58s | co-signing flags of the client. And so, if you're not familiar with hardened runtime, 850.58s - 855.58s | essentially what it is, is a co-signing flag that says to macOS to, uh, block all different 857.72s - 863.62s | manners of injecting, uh, code dynamically, and that makes sense, right? If your security 863.62s - 869.16s | model is going to be built around co-signing, you don't want ways to have untrusted code 869.16s - 874.10s | running in a process without invalidating that signature. And so that was introduced in 874.10s - 878.76s | September of 2018, and it would have been literally impossible for them to include that 878.76s - 883.78s | control, and they were a company before 2018. And so we thought, maybe if we could find 883.78s - 892.14s | some old clients, we could do this injection and satisfy their checks. Um, yeah, so like 894.20s - 898.24s | Jeff mentioned, what they were doing is checking the code signature information of the XPC 898.24s - 901.84s | client to see if it had a matching team identifier. They weren't checking for the 901.86s - 906.30s | hardened runtime, so we knew what we needed to find. An old version of 1Password that 906.30s - 911.78s | matched these conditions. Uh, we couldn't find any official source from 1Password that was 911.78s - 916.32s | offering old versions of clients, so we tried to get a little bit creative, and what we did 916.32s - 920.82s | is tried to enumerate subdomains of 1Password dot com by looking at certificate 920.82s - 926.62s | transparency logs. And one of the subdomains we found was c dot 1Password dot com, and it 926.62s - 931.30s | appeared


chunk_id: 216_chunk_18
chunk_text:
- 878.76s | September of 2018, and it would have been literally impossible for them to include that 878.76s - 883.78s | control, and they were a company before 2018. And so we thought, maybe if we could find 883.78s - 892.14s | some old clients, we could do this injection and satisfy their checks. Um, yeah, so like 894.20s - 898.24s | Jeff mentioned, what they were doing is checking the code signature information of the XPC 898.24s - 901.84s | client to see if it had a matching team identifier. They weren't checking for the 901.86s - 906.30s | hardened runtime, so we knew what we needed to find. An old version of 1Password that 906.30s - 911.78s | matched these conditions. Uh, we couldn't find any official source from 1Password that was 911.78s - 916.32s | offering old versions of clients, so we tried to get a little bit creative, and what we did 916.32s - 920.82s | is tried to enumerate subdomains of 1Password dot com by looking at certificate 920.82s - 926.62s | transparency logs. And one of the subdomains we found was c dot 1Password dot com, and it 926.62s - 931.30s | appeared to be some sort of web cache, we didn't really know what it was used for. Um, so we 931.32s - 937.00s | used Google dorks to see what files Google had indexed, um, on the URL, and we found a bunch 937.00s - 941.46s | of old versions of 1Password for all different operating systems. Um, one of the oldest ones 941.46s - 946.58s | we could find for Mac was version 6.5.2, so we pulled that down, and it looked like a 946.58s - 950.74s | perfect candidate because it had the correct team identifier, even though it was very old, um, 950.74s - 956.04s | and it didn't have the hardened runtime. So we figured we likely could inject into it. If 956.06s - 961.60s | you have a binary that is not signed with the hardened runtime, um, the easiest way to 961.60s - 967.20s | inject into it is to write your own dynamic library, uh, that runs whatever code you want in 967.20s - 972.34s | its constructor, and then you load that dynamic library, uh, as you launch the program using 972.34s - 977.34s | the dial D insert libraries environment variable. This


chunk_id: 216_chunk_19
chunk_text:
to be some sort of web cache, we didn't really know what it was used for. Um, so we 931.32s - 937.00s | used Google dorks to see what files Google had indexed, um, on the URL, and we found a bunch 937.00s - 941.46s | of old versions of 1Password for all different operating systems. Um, one of the oldest ones 941.46s - 946.58s | we could find for Mac was version 6.5.2, so we pulled that down, and it looked like a 946.58s - 950.74s | perfect candidate because it had the correct team identifier, even though it was very old, um, 950.74s - 956.04s | and it didn't have the hardened runtime. So we figured we likely could inject into it. If 956.06s - 961.60s | you have a binary that is not signed with the hardened runtime, um, the easiest way to 961.60s - 967.20s | inject into it is to write your own dynamic library, uh, that runs whatever code you want in 967.20s - 972.34s | its constructor, and then you load that dynamic library, uh, as you launch the program using 972.34s - 977.34s | the dial D insert libraries environment variable. This is very similar to LD preload on 977.34s - 982.22s | Linux, uh, for loading shared objects. Um, basically you can, it'll be the first dynamic 982.24s - 987.04s | library that the program loads, and it'll run your code pretty much immediately. So we knew 987.04s - 991.82s | that the XPC server exposed one method called echo, which just echoed a string back to you. 991.82s - 995.66s | So as a proof of concept, we just wrote our dynamic library to send the string hello world to 995.66s - 1000.76s | the server. Um, as you can see here, uh, once our code runs, it does send that string, we get a 1000.76s - 1005.76s | successful response, and this is great because it proves that, um, the XPC server validation 1005.76s - 1011.74s | was passed by our client. So obviously we didn't want to just echo strings back to 1011.76s - 1016.16s | ourselves, um, so we looked at what other methods were available, and the one that performed 1016.16s - 1020.54s | actually interesting functionality was called send to main, uh, where basically a message 1020.54s - 1025.50s | gets sent through browser helper


chunk_id: 216_chunk_20
chunk_text:
is very similar to LD preload on 977.34s - 982.22s | Linux, uh, for loading shared objects. Um, basically you can, it'll be the first dynamic 982.24s - 987.04s | library that the program loads, and it'll run your code pretty much immediately. So we knew 987.04s - 991.82s | that the XPC server exposed one method called echo, which just echoed a string back to you. 991.82s - 995.66s | So as a proof of concept, we just wrote our dynamic library to send the string hello world to 995.66s - 1000.76s | the server. Um, as you can see here, uh, once our code runs, it does send that string, we get a 1000.76s - 1005.76s | successful response, and this is great because it proves that, um, the XPC server validation 1005.76s - 1011.74s | was passed by our client. So obviously we didn't want to just echo strings back to 1011.76s - 1016.16s | ourselves, um, so we looked at what other methods were available, and the one that performed 1016.16s - 1020.54s | actually interesting functionality was called send to main, uh, where basically a message 1020.54s - 1025.50s | gets sent through browser helper and then to the main application, um, and gives you a 1025.50s - 1030.84s | response. Um, so what we tried doing was just sending the same messages that the browser 1030.84s - 1035.58s | extension was triggering, um, but unfortunately we were just getting no responses 1035.58s - 1040.52s | whatsoever when we would do that. It would just totally hang. Um, so through some debugging, 1040.54s - 1044.44s | we found that when our connection was created, there was a connection object and there was a 1044.44s - 1050.48s | boolean flag on it of is browser equals false, and we figured this might be problematic. Um, 1050.48s - 1055.18s | and it seemed to be getting set to false because our client's package ID, one password six in 1055.18s - 1060.36s | this case, uh, didn't match what was expected being the browser support process. Um, so we 1060.36s - 1065.42s | didn't give up there because we knew that the one password CLI tool, um, called OP, uh, we 1065.46s - 1070.54s | knew that that also communicated with the same XPC server using XPC messages, so we wanted to 1070.54s - 1076.94s | understand how


chunk_id: 216_chunk_21
chunk_text:
and then to the main application, um, and gives you a 1025.50s - 1030.84s | response. Um, so what we tried doing was just sending the same messages that the browser 1030.84s - 1035.58s | extension was triggering, um, but unfortunately we were just getting no responses 1035.58s - 1040.52s | whatsoever when we would do that. It would just totally hang. Um, so through some debugging, 1040.54s - 1044.44s | we found that when our connection was created, there was a connection object and there was a 1044.44s - 1050.48s | boolean flag on it of is browser equals false, and we figured this might be problematic. Um, 1050.48s - 1055.18s | and it seemed to be getting set to false because our client's package ID, one password six in 1055.18s - 1060.36s | this case, uh, didn't match what was expected being the browser support process. Um, so we 1060.36s - 1065.42s | didn't give up there because we knew that the one password CLI tool, um, called OP, uh, we 1065.46s - 1070.54s | knew that that also communicated with the same XPC server using XPC messages, so we wanted to 1070.54s - 1076.94s | understand how that worked. To do so, we pulled down this XPC spy, uh, open source program for 1076.94s - 1082.38s | inspecting XPC traffic that a process is sending, and we started using, uh, the OP tool, uh, 1082.38s - 1087.28s | while looking at the XPC traffic, and we saw that it was sending, uh, similar XPC traffic, but, 1087.28s - 1091.72s | um, it was including this client info JSON blob. Um, and I'll explain kind of the 1091.72s - 1096.40s | significance of some of the values you see, uh, highlighted in red here in the coming slides, 1096.40s - 1101.46s | but, um, the point is, we were able to send messages in the same way the one password CLI 1101.46s - 1106.80s | was and mimic the functionality of the one password CLI. And this was nice, um, but there's 1106.80s - 1112.44s | quite a few limitations that we weren't satisfied with. First, CLI integration is disabled by 1112.44s - 1117.48s | default. So, uh, if you're sending these messages and the CLI integration is disabled, nothing's 1117.48s - 1123.12s | gonna happen. Um, and, you know, most people probably haven't turned it on. So,


chunk_id: 216_chunk_22
chunk_text:
that worked. To do so, we pulled down this XPC spy, uh, open source program for 1076.94s - 1082.38s | inspecting XPC traffic that a process is sending, and we started using, uh, the OP tool, uh, 1082.38s - 1087.28s | while looking at the XPC traffic, and we saw that it was sending, uh, similar XPC traffic, but, 1087.28s - 1091.72s | um, it was including this client info JSON blob. Um, and I'll explain kind of the 1091.72s - 1096.40s | significance of some of the values you see, uh, highlighted in red here in the coming slides, 1096.40s - 1101.46s | but, um, the point is, we were able to send messages in the same way the one password CLI 1101.46s - 1106.80s | was and mimic the functionality of the one password CLI. And this was nice, um, but there's 1106.80s - 1112.44s | quite a few limitations that we weren't satisfied with. First, CLI integration is disabled by 1112.44s - 1117.48s | default. So, uh, if you're sending these messages and the CLI integration is disabled, nothing's 1117.48s - 1123.12s | gonna happen. Um, and, you know, most people probably haven't turned it on. So, we were 1123.12s - 1128.46s | wondering, could we just find where the settings are stored and enable it ourselves? And we 1128.46s - 1133.60s | found this settings dot JSON file. Um, we opened it, we saw a bunch of Boolean flags for various 1133.60s - 1138.36s | settings, one of them being CLI integration. Um, but we also saw that every setting had a 1138.36s - 1142.34s | corresponding auth tag, which looked like a big random string that we assumed was 1142.34s - 1146.34s | protecting the integrity of the settings through some authenticated hash or something like 1146.34s - 1151.40s | that. So, we assumed this wasn't gonna work, but we tried it anyway. Switched to CLI 1151.40s - 1155.34s | integration from false to true and it just worked. So, it seemed like these auth tags were 1155.34s - 1160.58s | here, probably secure, but they just weren't being enforced yet. So, we can enable CLI 1160.58s - 1166.62s | integration, great. Uh, but there's still, uh, some serious limitations here. Um, the first 1166.62s - 1171.00s | being that when you use the one password CLI, you have to approve the session. So, if


chunk_id: 216_chunk_23
chunk_text:
we were 1123.12s - 1128.46s | wondering, could we just find where the settings are stored and enable it ourselves? And we 1128.46s - 1133.60s | found this settings dot JSON file. Um, we opened it, we saw a bunch of Boolean flags for various 1133.60s - 1138.36s | settings, one of them being CLI integration. Um, but we also saw that every setting had a 1138.36s - 1142.34s | corresponding auth tag, which looked like a big random string that we assumed was 1142.34s - 1146.34s | protecting the integrity of the settings through some authenticated hash or something like 1146.34s - 1151.40s | that. So, we assumed this wasn't gonna work, but we tried it anyway. Switched to CLI 1151.40s - 1155.34s | integration from false to true and it just worked. So, it seemed like these auth tags were 1155.34s - 1160.58s | here, probably secure, but they just weren't being enforced yet. So, we can enable CLI 1160.58s - 1166.62s | integration, great. Uh, but there's still, uh, some serious limitations here. Um, the first 1166.62s - 1171.00s | being that when you use the one password CLI, you have to approve the session. So, if you've 1171.00s - 1174.30s | used it before, you're probably familiar with this image on the left, where it's gonna say 1174.36s - 1180.06s | your terminal is requesting access to one password, do you wanna authorize it or not? Um, and 1180.06s - 1186.64s | so what we saw was that your session basically gets created based on the XPC messages that 1186.64s - 1191.88s | get sent when you run the, uh, OP tool. And the session identifier is basically based on 1191.88s - 1196.88s | the PID of your shell, being the parent process that calls the tool, and the start time of 1196.88s - 1202.06s | that parent process. And so, it'd be pretty easy for malware on the machine to basically 1202.08s - 1207.66s | look for any shell that spawns the OP CLI, figure out the PID of that shell, and then grab the 1207.66s - 1211.26s | start time of the process, and then you'd know a valid session identifier and you could 1211.26s - 1216.26s | ride along with it, sending whatever messages you want with an approved session. However, 1216.26s - 1220.30s | um, this, you know, obviously isn't optimal either


chunk_id: 216_chunk_24
chunk_text:
you've 1171.00s - 1174.30s | used it before, you're probably familiar with this image on the left, where it's gonna say 1174.36s - 1180.06s | your terminal is requesting access to one password, do you wanna authorize it or not? Um, and 1180.06s - 1186.64s | so what we saw was that your session basically gets created based on the XPC messages that 1186.64s - 1191.88s | get sent when you run the, uh, OP tool. And the session identifier is basically based on 1191.88s - 1196.88s | the PID of your shell, being the parent process that calls the tool, and the start time of 1196.88s - 1202.06s | that parent process. And so, it'd be pretty easy for malware on the machine to basically 1202.08s - 1207.66s | look for any shell that spawns the OP CLI, figure out the PID of that shell, and then grab the 1207.66s - 1211.26s | start time of the process, and then you'd know a valid session identifier and you could 1211.26s - 1216.26s | ride along with it, sending whatever messages you want with an approved session. However, 1216.26s - 1220.30s | um, this, you know, obviously isn't optimal either because it requires that there's a 1220.30s - 1224.64s | legitimate user sitting there, really using the one password CLI, having an active session 1224.66s - 1229.66s | that you could spoof alongside. So we kept digging here, um, we, uh, saw that in the 1232.24s - 1236.90s | client info JSON blob, there was a session type of CLI being sent when the CLI was being 1236.90s - 1242.18s | used, and, uh, we were wondering if any other session types might be supported. We 1242.18s - 1247.08s | disassembled the browser helper binary to take a look, and we saw that it seemed like the 1247.08s - 1251.98s | string of browser could also be a valid session type. And this was interesting to us because 1252.00s - 1258.00s | we hadn't seen this string being sent by, uh, the extension. But, uh, we tried it anyway to 1258.00s - 1261.58s | see what would happen. We removed all of those other session identifiers that I was 1261.58s - 1266.78s | mentioning before, and it just totally worked. If one password was unlocked, we were able to 1266.78s - 1272.02s | send a session type browser, request the


chunk_id: 216_chunk_25
chunk_text:
because it requires that there's a 1220.30s - 1224.64s | legitimate user sitting there, really using the one password CLI, having an active session 1224.66s - 1229.66s | that you could spoof alongside. So we kept digging here, um, we, uh, saw that in the 1232.24s - 1236.90s | client info JSON blob, there was a session type of CLI being sent when the CLI was being 1236.90s - 1242.18s | used, and, uh, we were wondering if any other session types might be supported. We 1242.18s - 1247.08s | disassembled the browser helper binary to take a look, and we saw that it seemed like the 1247.08s - 1251.98s | string of browser could also be a valid session type. And this was interesting to us because 1252.00s - 1258.00s | we hadn't seen this string being sent by, uh, the extension. But, uh, we tried it anyway to 1258.00s - 1261.58s | see what would happen. We removed all of those other session identifiers that I was 1261.58s - 1266.78s | mentioning before, and it just totally worked. If one password was unlocked, we were able to 1266.78s - 1272.02s | send a session type browser, request the muck, and we were able to grab it. So that kind of 1272.02s - 1276.96s | completed the chain of, um, injecting into an old client, sending XPC messages, and grabbing 1276.98s - 1281.98s | the muck. Um, so you may have noticed that every issue we've discussed so far requires that 1284.08s - 1289.18s | one password is unlocked. Um, this next and final issue we're gonna discuss does not require 1289.18s - 1293.46s | that one password is unlocked. And it focuses on how biometric authentication is 1293.46s - 1298.26s | implemented. If you, uh, use one password on a MacBook, you're probably familiar with this. 1298.26s - 1303.80s | You launch one password, you provide your fingerprint, it unlocks. Um, so a very common way 1303.84s - 1308.28s | that macOS apps are gonna implement fingerprint checks is by using this evaluate policy 1308.28s - 1313.52s | method. And it's pretty simple. You call it, prompts user for their fingerprint. If they 1313.52s - 1318.18s | provide it and it's valid, it returns true, otherwise it returns false. So what we did to make 1318.18s - 1322.92s | sure this method was in use, we hooked it


chunk_id: 216_chunk_26
chunk_text:
muck, and we were able to grab it. So that kind of 1272.02s - 1276.96s | completed the chain of, um, injecting into an old client, sending XPC messages, and grabbing 1276.98s - 1281.98s | the muck. Um, so you may have noticed that every issue we've discussed so far requires that 1284.08s - 1289.18s | one password is unlocked. Um, this next and final issue we're gonna discuss does not require 1289.18s - 1293.46s | that one password is unlocked. And it focuses on how biometric authentication is 1293.46s - 1298.26s | implemented. If you, uh, use one password on a MacBook, you're probably familiar with this. 1298.26s - 1303.80s | You launch one password, you provide your fingerprint, it unlocks. Um, so a very common way 1303.84s - 1308.28s | that macOS apps are gonna implement fingerprint checks is by using this evaluate policy 1308.28s - 1313.52s | method. And it's pretty simple. You call it, prompts user for their fingerprint. If they 1313.52s - 1318.18s | provide it and it's valid, it returns true, otherwise it returns false. So what we did to make 1318.18s - 1322.92s | sure this method was in use, we hooked it with Frida, uh, used the code on the right, uh, 1322.92s - 1328.10s | screenshot here, and just basically made it always return true no matter what. And so after 1328.10s - 1332.54s | doing that, we launch one password, prompt it for our fingerprint. We could either hit cancel 1332.54s - 1337.54s | or provide someone else's fingerprint, and the vault unlocks. So, um, you might be 1339.72s - 1342.98s | thinking, well, you know, if you're a red team or you land on someone's laptop, you can't just 1342.98s - 1347.16s | disable SIP and hook methods with Frida. And that's true, but this taught us something 1347.16s - 1351.76s | really interesting, which is that even with an invalid fingerprint, somehow one password is 1351.76s - 1355.70s | grabbing the muck and it is decrypting the vaults and displaying your passwords. So we wanted 1355.70s - 1360.96s | to understand how that worked. Um, and also it's not obvious that this was gonna work just 1360.96s - 1366.50s | hooking, um, methods with Frida, cause it is possible to protect items at the keychain level 1366.50s - 1370.14s | with flags that indicate a fingerprint is required.


chunk_id: 216_chunk_27
chunk_text:
with Frida, uh, used the code on the right, uh, 1322.92s - 1328.10s | screenshot here, and just basically made it always return true no matter what. And so after 1328.10s - 1332.54s | doing that, we launch one password, prompt it for our fingerprint. We could either hit cancel 1332.54s - 1337.54s | or provide someone else's fingerprint, and the vault unlocks. So, um, you might be 1339.72s - 1342.98s | thinking, well, you know, if you're a red team or you land on someone's laptop, you can't just 1342.98s - 1347.16s | disable SIP and hook methods with Frida. And that's true, but this taught us something 1347.16s - 1351.76s | really interesting, which is that even with an invalid fingerprint, somehow one password is 1351.76s - 1355.70s | grabbing the muck and it is decrypting the vaults and displaying your passwords. So we wanted 1355.70s - 1360.96s | to understand how that worked. Um, and also it's not obvious that this was gonna work just 1360.96s - 1366.50s | hooking, um, methods with Frida, cause it is possible to protect items at the keychain level 1366.50s - 1370.14s | with flags that indicate a fingerprint is required. And you wouldn't be able to just hook 1370.14s - 1373.74s | something with Frida and make that work, cause the keychain would protect you with your 1373.74s - 1379.30s | fingerprint. So that was not what was going on here. So how does it work? We figured the 1379.30s - 1383.38s | keychain was involved in some way, so we opened the keychain access application and started 1383.38s - 1388.28s | looking around, and we saw that when using biometric authentication, there were, uh, these 1388.30s - 1392.44s | items, or there was a item in the keychain called biometric unlock, and it would be named 1392.44s - 1396.38s | differently depending on if you were using SSO or not. And we saw that it could only be 1396.38s - 1400.48s | accessed by one password, so if it wasn't obvious from the name, we knew it was, uh, being 1400.48s - 1406.76s | used by one password for biometric unlock. Now, along with, um, the item in the keychain, 1406.76s - 1410.60s | there were these comments that were added, which tell us that it's, the item here is 1410.60s - 1415.36s | encrypted by the secure enclave.


chunk_id: 216_chunk_28
chunk_text:
And you wouldn't be able to just hook 1370.14s - 1373.74s | something with Frida and make that work, cause the keychain would protect you with your 1373.74s - 1379.30s | fingerprint. So that was not what was going on here. So how does it work? We figured the 1379.30s - 1383.38s | keychain was involved in some way, so we opened the keychain access application and started 1383.38s - 1388.28s | looking around, and we saw that when using biometric authentication, there were, uh, these 1388.30s - 1392.44s | items, or there was a item in the keychain called biometric unlock, and it would be named 1392.44s - 1396.38s | differently depending on if you were using SSO or not. And we saw that it could only be 1396.38s - 1400.48s | accessed by one password, so if it wasn't obvious from the name, we knew it was, uh, being 1400.48s - 1406.76s | used by one password for biometric unlock. Now, along with, um, the item in the keychain, 1406.76s - 1410.60s | there were these comments that were added, which tell us that it's, the item here is 1410.60s - 1415.36s | encrypted by the secure enclave. And this tracked because when we looked at what was stored 1415.38s - 1420.56s | in the item, it was just a bunch of random ciphertext that we couldn't do anything with. So, 1420.56s - 1424.12s | we were wondering where the secure enclave key entry is, because we couldn't see it in the 1424.12s - 1428.80s | keychain access application, we couldn't query it with the security CLI tool, which is the 1428.80s - 1432.94s | built-in, uh, macOS CLI tool for interacting with the keychain. Uh, so we wanted to 1432.94s - 1438.14s | understand where it was, what was going on and how it was being used. What we did for this 1438.14s - 1443.44s | was hooked the sec item copy matching, uh, method with Frida, and logged the parameters 1443.46s - 1448.46s | passed to it and the return values from it. This method is used to query the keychain. And so 1448.46s - 1452.54s | we wanted to trigger the fingerprint unlock and see what was, uh, being accessed from the 1452.54s - 1457.70s | keychain by one password when going through that flow. And amongst other things, one of the 1457.70s - 1464.38s


chunk_id: 216_chunk_29
chunk_text:
And this tracked because when we looked at what was stored 1415.38s - 1420.56s | in the item, it was just a bunch of random ciphertext that we couldn't do anything with. So, 1420.56s - 1424.12s | we were wondering where the secure enclave key entry is, because we couldn't see it in the 1424.12s - 1428.80s | keychain access application, we couldn't query it with the security CLI tool, which is the 1428.80s - 1432.94s | built-in, uh, macOS CLI tool for interacting with the keychain. Uh, so we wanted to 1432.94s - 1438.14s | understand where it was, what was going on and how it was being used. What we did for this 1438.14s - 1443.44s | was hooked the sec item copy matching, uh, method with Frida, and logged the parameters 1443.46s - 1448.46s | passed to it and the return values from it. This method is used to query the keychain. And so 1448.46s - 1452.54s | we wanted to trigger the fingerprint unlock and see what was, uh, being accessed from the 1452.54s - 1457.70s | keychain by one password when going through that flow. And amongst other things, one of the 1457.70s - 1464.38s | things we saw accessed was a sec key ref or a secret key reference. And, um, that's what was 1464.38s - 1469.46s | returned. And in the query was an A tag or an application tag, uh, with those hex bytes you 1469.48s - 1474.98s | see in the top red box. And if you decode those, you get the string of com agile bits one 1474.98s - 1479.58s | password memory enclave key. So this seemed likely what we were, uh, looking for based on 1479.58s - 1485.92s | the name, but we wanted to be sure. So, uh, we wanted to see if it was being used to decrypt 1485.92s - 1491.06s | anything or decrypt that biometric unlock entry. Um, now secure enclave keys, it's not the 1491.06s - 1495.36s | case that you just access them and then do what you wish with the bytes. Instead, you ask the 1495.36s - 1500.14s | secure enclave to encrypt something, decrypt something, or sign something. In this case, it, we 1500.14s - 1504.40s | thought it would be decrypting something. So we did the same thing with hooking sec key create 1504.40s - 1509.42s | decrypted data,


chunk_id: 216_chunk_30
chunk_text:
| things we saw accessed was a sec key ref or a secret key reference. And, um, that's what was 1464.38s - 1469.46s | returned. And in the query was an A tag or an application tag, uh, with those hex bytes you 1469.48s - 1474.98s | see in the top red box. And if you decode those, you get the string of com agile bits one 1474.98s - 1479.58s | password memory enclave key. So this seemed likely what we were, uh, looking for based on 1479.58s - 1485.92s | the name, but we wanted to be sure. So, uh, we wanted to see if it was being used to decrypt 1485.92s - 1491.06s | anything or decrypt that biometric unlock entry. Um, now secure enclave keys, it's not the 1491.06s - 1495.36s | case that you just access them and then do what you wish with the bytes. Instead, you ask the 1495.36s - 1500.14s | secure enclave to encrypt something, decrypt something, or sign something. In this case, it, we 1500.14s - 1504.40s | thought it would be decrypting something. So we did the same thing with hooking sec key create 1504.40s - 1509.42s | decrypted data, which is how you would ask the secure enclave to decrypt something. And we log 1509.42s - 1514.28s | the parameters and the return value. And we do see that that's during the biometric unlock, that 1514.28s - 1518.72s | sec key ref is being used. It's decrypting the ciphertext that was stored in the biometric 1518.72s - 1524.16s | unlock entry. And in return, we get 32 bytes. Uh, so we thought 32 bytes, that's great. That 1524.16s - 1529.94s | sounds like an AES key. Maybe that's the muck. But it turned out it was not the muck. And we 1529.94s - 1533.84s | didn't really know what to do. So we just took these 32 bytes and started trial and error 1533.84s - 1538.84s | trying to decrypt things. And, um, finally, we found an item in the encrypted SQLite database 1541.32s - 1548.02s | called enc unlock key. And we were able to successfully decrypt it with AES. And, uh, in 1548.02s - 1553.36s | return, we got the muck. So, now we fully understood the chain here of the memory 1553.36s - 1557.76s | enclave key in the secure enclave being used to decrypt the


chunk_id: 216_chunk_31
chunk_text:
which is how you would ask the secure enclave to decrypt something. And we log 1509.42s - 1514.28s | the parameters and the return value. And we do see that that's during the biometric unlock, that 1514.28s - 1518.72s | sec key ref is being used. It's decrypting the ciphertext that was stored in the biometric 1518.72s - 1524.16s | unlock entry. And in return, we get 32 bytes. Uh, so we thought 32 bytes, that's great. That 1524.16s - 1529.94s | sounds like an AES key. Maybe that's the muck. But it turned out it was not the muck. And we 1529.94s - 1533.84s | didn't really know what to do. So we just took these 32 bytes and started trial and error 1533.84s - 1538.84s | trying to decrypt things. And, um, finally, we found an item in the encrypted SQLite database 1541.32s - 1548.02s | called enc unlock key. And we were able to successfully decrypt it with AES. And, uh, in 1548.02s - 1553.36s | return, we got the muck. So, now we fully understood the chain here of the memory 1553.36s - 1557.76s | enclave key in the secure enclave being used to decrypt the biometric unlock entry in the 1557.76s - 1562.06s | keychain. From there, you have a key that can be used to decrypt this entry in the SQLite 1562.06s - 1569.08s | database. Then you have the muck, which can decrypt the rest. Um, so now we fully understand 1569.08s - 1574.38s | this process, but obviously we can't just access the keychain, um, on behalf of 1Password 1574.38s - 1578.88s | unless we have a way to inject our own code into 1Password and query the keychain. Because 1578.88s - 1582.48s | those items can only be queried by 1Password. So now we're gonna talk about how we 1582.50s - 1590.85s | accomplished getting that code injection. Yeah, so we needed to, uh, have the 1Password 1590.85s - 1597.09s | bundle ID and we needed to not invalidate the code signature. And, uh, there's a relatively 1597.09s - 1602.75s | old technique on Electron where you can inject, uh, essentially the main.js file isn't 1602.75s - 1607.47s | included in the signature check and so you can just overwrite it. And to protect against 1607.47s - 1612.93s | that, they added this enable embedded ASAR integrity validation fuse. And this


chunk_id: 216_chunk_32
chunk_text:
biometric unlock entry in the 1557.76s - 1562.06s | keychain. From there, you have a key that can be used to decrypt this entry in the SQLite 1562.06s - 1569.08s | database. Then you have the muck, which can decrypt the rest. Um, so now we fully understand 1569.08s - 1574.38s | this process, but obviously we can't just access the keychain, um, on behalf of 1Password 1574.38s - 1578.88s | unless we have a way to inject our own code into 1Password and query the keychain. Because 1578.88s - 1582.48s | those items can only be queried by 1Password. So now we're gonna talk about how we 1582.50s - 1590.85s | accomplished getting that code injection. Yeah, so we needed to, uh, have the 1Password 1590.85s - 1597.09s | bundle ID and we needed to not invalidate the code signature. And, uh, there's a relatively 1597.09s - 1602.75s | old technique on Electron where you can inject, uh, essentially the main.js file isn't 1602.75s - 1607.47s | included in the signature check and so you can just overwrite it. And to protect against 1607.47s - 1612.93s | that, they added this enable embedded ASAR integrity validation fuse. And this was enabled 1612.95s - 1617.45s | in the modern version of 1Password. Uh, but we implied the same thinking that we did to the 1617.45s - 1621.23s | hardened runtime and went through all of our scraped old versions. And we found one that 1621.23s - 1625.63s | was created either before or around the time when this flag was introduced and it had it 1625.63s - 1630.31s | disabled. Even if we couldn't find that though, there was this additional GitHub security 1630.31s - 1634.67s | advisory that was a bypass to this fuse. And, uh, you know, if we couldn't find either of 1634.67s - 1640.79s | those, we probably would have gone and hunted for some similar bug ourselves. So we can 1640.81s - 1645.95s | overwrite the main.js file, but that isn't good enough to do what we need to do. Uh, 1645.95s - 1649.51s | essentially there are no functions in Node.js that we can call directly that will let us 1649.51s - 1653.93s | read the keychain and use the secure enclave. And so we need to call these native functions 1653.93s - 1658.59s | from native code. And because of the hardened runtime,


chunk_id: 216_chunk_33
chunk_text:
was enabled 1612.95s - 1617.45s | in the modern version of 1Password. Uh, but we implied the same thinking that we did to the 1617.45s - 1621.23s | hardened runtime and went through all of our scraped old versions. And we found one that 1621.23s - 1625.63s | was created either before or around the time when this flag was introduced and it had it 1625.63s - 1630.31s | disabled. Even if we couldn't find that though, there was this additional GitHub security 1630.31s - 1634.67s | advisory that was a bypass to this fuse. And, uh, you know, if we couldn't find either of 1634.67s - 1640.79s | those, we probably would have gone and hunted for some similar bug ourselves. So we can 1640.81s - 1645.95s | overwrite the main.js file, but that isn't good enough to do what we need to do. Uh, 1645.95s - 1649.51s | essentially there are no functions in Node.js that we can call directly that will let us 1649.51s - 1653.93s | read the keychain and use the secure enclave. And so we need to call these native functions 1653.93s - 1658.59s | from native code. And because of the hardened runtime, which is applied to this version of 1658.59s - 1662.87s | 1Password, we can't actually do the typical thing where you'd write your own library and 1662.87s - 1668.37s | have the Node.js code hand off to it. And so we took a step back and saw that in the 1668.39s - 1673.79s | entitlements, uh, there was this Apple security allow JIT entitlement, which essentially 1673.79s - 1677.83s | means that somewhere in the process it would be okay to have a read write execute page of 1677.83s - 1683.57s | memory. Once we saw that, we realized a browser exploit would probably work. And so we 1683.57s - 1688.27s | found an applicable CVE that was exploited in the wild. We did the standard browser exploit 1688.27s - 1693.21s | things and got our JavaScript heap and read write primitives and used that to run shellcode 1693.21s - 1697.79s | by corrupting a WASM object. I'll show you a video of that in a second. But I also wanted 1697.81s - 1701.57s | to call out that because we're not in a renderer process, we can kind of cheat a little bit 1701.57s - 1706.17s | with ASLR


chunk_id: 216_chunk_34
chunk_text:
which is applied to this version of 1658.59s - 1662.87s | 1Password, we can't actually do the typical thing where you'd write your own library and 1662.87s - 1668.37s | have the Node.js code hand off to it. And so we took a step back and saw that in the 1668.39s - 1673.79s | entitlements, uh, there was this Apple security allow JIT entitlement, which essentially 1673.79s - 1677.83s | means that somewhere in the process it would be okay to have a read write execute page of 1677.83s - 1683.57s | memory. Once we saw that, we realized a browser exploit would probably work. And so we 1683.57s - 1688.27s | found an applicable CVE that was exploited in the wild. We did the standard browser exploit 1688.27s - 1693.21s | things and got our JavaScript heap and read write primitives and used that to run shellcode 1693.21s - 1697.79s | by corrupting a WASM object. I'll show you a video of that in a second. But I also wanted 1697.81s - 1701.57s | to call out that because we're not in a renderer process, we can kind of cheat a little bit 1701.57s - 1706.17s | with ASLR by just running VM map outside of the process and then directly writing all 1706.17s - 1722.49s | the addresses we need to a file and then reading that file into our exploit. Alright, so 1722.49s - 1728.17s | hopefully you see that, it's on the monitor. Yeah, so in this video we're just adding a 1728.17s - 1732.91s | demo password that we're gonna extract. Uh, we're adding it right now, then we'll show you 1732.91s - 1737.65s | that SIP is enabled. And then what we're gonna do is run a makefile. And that makefile is 1737.65s - 1741.83s | gonna compile a C program that prints out some function addresses in the core foundation 1741.83s - 1746.27s | library. The reason this is important is because we need those functions, uh, but we didn't 1746.27s - 1750.63s | want to do the work of adjusting the ASLR ourselves. And these functions are in the shared 1750.63s - 1755.37s | cache, so we can just print them out and the addresses are the same between processes. That 1755.37s - 1761.51s | print is gonna be used to generate our final ASLR adjusted, uh, shellcode in JavaScript. 1761.51s -


chunk_id: 216_chunk_35
chunk_text:
by just running VM map outside of the process and then directly writing all 1706.17s - 1722.49s | the addresses we need to a file and then reading that file into our exploit. Alright, so 1722.49s - 1728.17s | hopefully you see that, it's on the monitor. Yeah, so in this video we're just adding a 1728.17s - 1732.91s | demo password that we're gonna extract. Uh, we're adding it right now, then we'll show you 1732.91s - 1737.65s | that SIP is enabled. And then what we're gonna do is run a makefile. And that makefile is 1737.65s - 1741.83s | gonna compile a C program that prints out some function addresses in the core foundation 1741.83s - 1746.27s | library. The reason this is important is because we need those functions, uh, but we didn't 1746.27s - 1750.63s | want to do the work of adjusting the ASLR ourselves. And these functions are in the shared 1750.63s - 1755.37s | cache, so we can just print them out and the addresses are the same between processes. That 1755.37s - 1761.51s | print is gonna be used to generate our final ASLR adjusted, uh, shellcode in JavaScript. 1761.51s - 1767.15s | And we're gonna start the old version of 1Password. The reason we do that is because on the 1767.15s - 1771.85s | first run, a deep signature check is gonna be performed. Uh, but we can just start it, kill 1771.85s - 1777.93s | it, and then run it again with our modified main.js. And because this fuse is disabled and 1777.93s - 1785.57s | only a shallow check runs, we can run our browser exploit. Yeah, and so there it is. We'll 1785.57s - 1823.73s | show you that the password matches. I think it's up. No, I'm not good. Oh man, this is the 1823.73s - 1833.28s | hardest part of the talk. There we are. Okay, so this is the doc- oh man. Okay, so this is 1837.18s - 1840.88s | the diagram of what we essentially did. Uh, and I think it's important to call out here that 1840.88s - 1846.02s | although obviously 1Password could have used that enforcement flag, uh, but for everything else 1846.02s - 1850.22s | in order to get that native code execution and actually exploit it, we essentially just 1850.22s - 1854.46s | waited. And you know, we


chunk_id: 216_chunk_36
chunk_text:
1767.15s | And we're gonna start the old version of 1Password. The reason we do that is because on the 1767.15s - 1771.85s | first run, a deep signature check is gonna be performed. Uh, but we can just start it, kill 1771.85s - 1777.93s | it, and then run it again with our modified main.js. And because this fuse is disabled and 1777.93s - 1785.57s | only a shallow check runs, we can run our browser exploit. Yeah, and so there it is. We'll 1785.57s - 1823.73s | show you that the password matches. I think it's up. No, I'm not good. Oh man, this is the 1823.73s - 1833.28s | hardest part of the talk. There we are. Okay, so this is the doc- oh man. Okay, so this is 1837.18s - 1840.88s | the diagram of what we essentially did. Uh, and I think it's important to call out here that 1840.88s - 1846.02s | although obviously 1Password could have used that enforcement flag, uh, but for everything else 1846.02s - 1850.22s | in order to get that native code execution and actually exploit it, we essentially just 1850.22s - 1854.46s | waited. And you know, we scraped the old versions, but the vulnerabilities were in Electron 1854.46s - 1858.26s | and- and Chromium and it can be really hard to secure that when you're releasing new versions, 1858.26s - 1865.32s | uh, every day. And so here's a vulnerability summary of what we reported. Uh, this is a 1865.34s - 1869.72s | little bit, uh, there are a few more vulnerabilities here than what's going to be on the next 1869.72s - 1874.88s | slide. The reason that is, is because the initial XPC entry vulnerability is really what 1874.88s - 1879.50s | needed to be fixed, not the other stuff that we talked about with the session writing. And we 1879.50s - 1886.61s | also reported the, uh, Chrome managed policy bypasses to them, or to Google rather. And so 1886.61s - 1890.81s | here's an exploitability summary. Uh, I think it's important to call out that SIP doesn't 1890.81s - 1896.05s | prevent this and root is not required. Low privilege malware can perform these attacks. Uh, 1896.07s - 1902.51s | but, uh, the main issues that 1Password was able to fix are- are highlighted in green. And 1902.51s - 1908.05s | so the


chunk_id: 216_chunk_37
chunk_text:
scraped the old versions, but the vulnerabilities were in Electron 1854.46s - 1858.26s | and- and Chromium and it can be really hard to secure that when you're releasing new versions, 1858.26s - 1865.32s | uh, every day. And so here's a vulnerability summary of what we reported. Uh, this is a 1865.34s - 1869.72s | little bit, uh, there are a few more vulnerabilities here than what's going to be on the next 1869.72s - 1874.88s | slide. The reason that is, is because the initial XPC entry vulnerability is really what 1874.88s - 1879.50s | needed to be fixed, not the other stuff that we talked about with the session writing. And we 1879.50s - 1886.61s | also reported the, uh, Chrome managed policy bypasses to them, or to Google rather. And so 1886.61s - 1890.81s | here's an exploitability summary. Uh, I think it's important to call out that SIP doesn't 1890.81s - 1896.05s | prevent this and root is not required. Low privilege malware can perform these attacks. Uh, 1896.07s - 1902.51s | but, uh, the main issues that 1Password was able to fix are- are highlighted in green. And 1902.51s - 1908.05s | so the XPC bypass was fixed, the biometric enforcement flag was fixed, and most 1908.05s - 1912.63s | importantly, the settings modification is now being enforced. And you might say, well, why 1912.63s - 1917.83s | is that important? It's important because the first two currently exploitable issues are not 1917.83s - 1921.79s | exploitable if you just disable that setting. And so while it might degrade the user 1921.81s - 1927.51s | experience a little bit, you are fully protected with the newest version of 8.10.38 which was 1927.51s - 1932.25s | released in August. Now the- the browser extension injection and impersonation, like I 1932.25s - 1937.75s | mentioned earlier, that's not really a- a bug per se, it's kind of just how browsers work. And 1937.75s - 1941.93s | the browser support get PBID bypass is a little bit interesting because there's not really a 1941.93s - 1947.87s | clear line of ownership to the issue. Uh, so 1Password can't really securely look up the code 1947.91s - 1954.35s | signing information of their parent process. It is true that all browsers could maybe switch to 1954.35s - 1959.39s | XPC to provide that audit token. Uh, it's one


chunk_id: 216_chunk_38
chunk_text:
XPC bypass was fixed, the biometric enforcement flag was fixed, and most 1908.05s - 1912.63s | importantly, the settings modification is now being enforced. And you might say, well, why 1912.63s - 1917.83s | is that important? It's important because the first two currently exploitable issues are not 1917.83s - 1921.79s | exploitable if you just disable that setting. And so while it might degrade the user 1921.81s - 1927.51s | experience a little bit, you are fully protected with the newest version of 8.10.38 which was 1927.51s - 1932.25s | released in August. Now the- the browser extension injection and impersonation, like I 1932.25s - 1937.75s | mentioned earlier, that's not really a- a bug per se, it's kind of just how browsers work. And 1937.75s - 1941.93s | the browser support get PBID bypass is a little bit interesting because there's not really a 1941.93s - 1947.87s | clear line of ownership to the issue. Uh, so 1Password can't really securely look up the code 1947.91s - 1954.35s | signing information of their parent process. It is true that all browsers could maybe switch to 1954.35s - 1959.39s | XPC to provide that audit token. Uh, it's one available method to perform inter-process 1959.39s - 1964.79s | communication with, uh, or on Mac OS rather. But it's also true that Apple might be able to 1964.79s - 1970.63s | provide some way to look up your parent processes code signing information securely. It's 1970.63s - 1978.49s | just not super clear who, uh, who needs to go and fix this. And so, because you can't really 1978.49s - 1982.43s | fix those things, we're going to talk about how to detect them. So, I pulled all of these 1982.43s - 1987.37s | bundle IDs from the browser support process. These are all the chromium based browsers that 1987.37s - 1992.51s | it will accept as a parent process. And these are the non-chromium based browsers that it will 1992.51s - 1998.25s | accept as a parent process. And so, for the, uh, extension, injection, and impersonation, 1998.25s - 2002.11s | those are relatively straightforward to fix. Essentially, what you have to do is go 2002.11s - 2006.55s | enumerate the dangerous flags, which I've done for you. They're highlighted in orange. And 2006.55s - 2010.79s | you just need to alert any time those flags are used. Uh,


chunk_id: 216_chunk_39
chunk_text:
available method to perform inter-process 1959.39s - 1964.79s | communication with, uh, or on Mac OS rather. But it's also true that Apple might be able to 1964.79s - 1970.63s | provide some way to look up your parent processes code signing information securely. It's 1970.63s - 1978.49s | just not super clear who, uh, who needs to go and fix this. And so, because you can't really 1978.49s - 1982.43s | fix those things, we're going to talk about how to detect them. So, I pulled all of these 1982.43s - 1987.37s | bundle IDs from the browser support process. These are all the chromium based browsers that 1987.37s - 1992.51s | it will accept as a parent process. And these are the non-chromium based browsers that it will 1992.51s - 1998.25s | accept as a parent process. And so, for the, uh, extension, injection, and impersonation, 1998.25s - 2002.11s | those are relatively straightforward to fix. Essentially, what you have to do is go 2002.11s - 2006.55s | enumerate the dangerous flags, which I've done for you. They're highlighted in orange. And 2006.55s - 2010.79s | you just need to alert any time those flags are used. Uh, we also, out of an abundance of 2010.79s - 2014.23s | caution, are just blocking the developer builds of certain browsers because they have a 2014.23s - 2019.93s | slightly different threat model. And for the browser support issue, again, if you just disable 2019.93s - 2025.01s | that setting, uh, you will be fine. Uh, I can tell you personally, I'm not going to disable 2025.01s - 2029.41s | the setting. I like using the browser extension. And I think having malware on your computer 2029.41s - 2034.81s | is already a pretty significant bar. Now, uh, if you don't want to disable the extension like 2034.81s - 2040.09s | me, here's a, you can alert on it. Uh, basically, during normal use, when you click a browser 2040.09s - 2045.65s | icon, what's going to happen is launch D actually spawns that process. And so, if you just set 2045.65s - 2050.67s | an alert for any time a browser, uh, is spawned, and the parent is not launch D, it's likely 2052.79s - 2057.57s | malicious, or it's going to be some kind of headless browsing. And you can usually limit 2057.57s - 2063.91s | that to


chunk_id: 216_chunk_40
chunk_text:
we also, out of an abundance of 2010.79s - 2014.23s | caution, are just blocking the developer builds of certain browsers because they have a 2014.23s - 2019.93s | slightly different threat model. And for the browser support issue, again, if you just disable 2019.93s - 2025.01s | that setting, uh, you will be fine. Uh, I can tell you personally, I'm not going to disable 2025.01s - 2029.41s | the setting. I like using the browser extension. And I think having malware on your computer 2029.41s - 2034.81s | is already a pretty significant bar. Now, uh, if you don't want to disable the extension like 2034.81s - 2040.09s | me, here's a, you can alert on it. Uh, basically, during normal use, when you click a browser 2040.09s - 2045.65s | icon, what's going to happen is launch D actually spawns that process. And so, if you just set 2045.65s - 2050.67s | an alert for any time a browser, uh, is spawned, and the parent is not launch D, it's likely 2052.79s - 2057.57s | malicious, or it's going to be some kind of headless browsing. And you can usually limit 2057.57s - 2063.91s | that to the people that need to do that at your company. Probably QA teams. And a malicious 2063.91s - 2068.65s | actor might look at that alert and, and try to get around it, right? By getting launch D to 2068.65s - 2073.45s | spawn their parent, uh, to spawn their exploit process and then execing in directly. And so 2073.45s - 2078.33s | what you can look for there is the exec. Uh, the, the point of looking for the exec is that 2078.33s - 2082.27s | there are a couple different ways to obscure or, or, you know, move around those read write 2082.27s - 2088.37s | pipes. You don't have to use the process tree that we showed you here. But at every, um, for 2088.37s - 2093.57s | every actual exploit, at some point, they're going to have to exec into a trusted browser. And 2093.57s - 2098.51s | so, that's what you will, uh, would alert on and it looks like PID reuse in launch D child 2098.51s - 2104.21s | processes. So here's our full disclosure timeline. We reported all the stuff that we found 2104.21s - 2110.19s | initially in January of 2024. And


chunk_id: 216_chunk_41
chunk_text:
the people that need to do that at your company. Probably QA teams. And a malicious 2063.91s - 2068.65s | actor might look at that alert and, and try to get around it, right? By getting launch D to 2068.65s - 2073.45s | spawn their parent, uh, to spawn their exploit process and then execing in directly. And so 2073.45s - 2078.33s | what you can look for there is the exec. Uh, the, the point of looking for the exec is that 2078.33s - 2082.27s | there are a couple different ways to obscure or, or, you know, move around those read write 2082.27s - 2088.37s | pipes. You don't have to use the process tree that we showed you here. But at every, um, for 2088.37s - 2093.57s | every actual exploit, at some point, they're going to have to exec into a trusted browser. And 2093.57s - 2098.51s | so, that's what you will, uh, would alert on and it looks like PID reuse in launch D child 2098.51s - 2104.21s | processes. So here's our full disclosure timeline. We reported all the stuff that we found 2104.21s - 2110.19s | initially in January of 2024. And then we followed up on March 8th and reported, reported 2110.19s - 2116.13s | that the biometrics flag was missing. And we actually weren't convinced at the time that it 2116.13s - 2120.33s | was going to be exploitable in practice. And so we put together a proof of concept and gave 2120.35s - 2124.47s | it to them on March 22nd and that confirmed for both us and 1Password that this was going to 2124.47s - 2129.47s | work. Here's a remediation timeline. Again, the, the browser support fix is unclear, but you 2131.63s - 2137.67s | can toggle that setting. The XPC bypass was fixed in April. Uh, the settings file 2137.67s - 2141.77s | modifications, this says it was scheduled to be fixed in August. That version is out now. We 2141.77s - 2148.19s | just submitted the slides before. And the biometrics flag issue was fixed in 8.10.36 and 2148.19s - 2153.99s | they followed up in 8.10.38 with just a reliability change. It was actually fixed in 3.6, 2153.99s - 2162.65s | but I would just recommend upgrading to the, to the latest version. So here is where you can 2162.65s - 2167.65s | find


chunk_id: 216_chunk_42
chunk_text:
then we followed up on March 8th and reported, reported 2110.19s - 2116.13s | that the biometrics flag was missing. And we actually weren't convinced at the time that it 2116.13s - 2120.33s | was going to be exploitable in practice. And so we put together a proof of concept and gave 2120.35s - 2124.47s | it to them on March 22nd and that confirmed for both us and 1Password that this was going to 2124.47s - 2129.47s | work. Here's a remediation timeline. Again, the, the browser support fix is unclear, but you 2131.63s - 2137.67s | can toggle that setting. The XPC bypass was fixed in April. Uh, the settings file 2137.67s - 2141.77s | modifications, this says it was scheduled to be fixed in August. That version is out now. We 2141.77s - 2148.19s | just submitted the slides before. And the biometrics flag issue was fixed in 8.10.36 and 2148.19s - 2153.99s | they followed up in 8.10.38 with just a reliability change. It was actually fixed in 3.6, 2153.99s - 2162.65s | but I would just recommend upgrading to the, to the latest version. So here is where you can 2162.65s - 2167.65s | find the code we've released at, uh, Morgan C3 slash 1Password exploits. Um, as of now we 2169.89s - 2174.69s | released, uh, the Frida scripts we used while doing some of this research. Um, none of it is 2174.69s - 2179.47s | really specific to 1Password at all. Um, it could definitely be extended and useful for 2179.49s - 2184.63s | researching other applications. In terms of exploits, the only thing we've released is a proof 2184.63s - 2189.87s | of concept for the XPC validation bypass issue. It's not going to decrypt your vaults, but 2189.87s - 2195.91s | rather just demonstrate the issue. Uh, in the future, we, uh, will add more code here once ample 2195.91s - 2200.29s | time has been given for people to be upgrading their 1Password to safe versions and for the 2200.29s - 2207.12s | other issues to be fixed. Um, and yeah, we may have some time for a few questions if people 2207.20s - 2239.27s | have them. Sorry, what was that? Yeah, we, we took a look at them. Um, I would just say any, 2239.27s - 2253.36s | okay, sounds good. Yeah. Any other password managers? Yes. It


chunk_id: 216_chunk_43
chunk_text:
the code we've released at, uh, Morgan C3 slash 1Password exploits. Um, as of now we 2169.89s - 2174.69s | released, uh, the Frida scripts we used while doing some of this research. Um, none of it is 2174.69s - 2179.47s | really specific to 1Password at all. Um, it could definitely be extended and useful for 2179.49s - 2184.63s | researching other applications. In terms of exploits, the only thing we've released is a proof 2184.63s - 2189.87s | of concept for the XPC validation bypass issue. It's not going to decrypt your vaults, but 2189.87s - 2195.91s | rather just demonstrate the issue. Uh, in the future, we, uh, will add more code here once ample 2195.91s - 2200.29s | time has been given for people to be upgrading their 1Password to safe versions and for the 2200.29s - 2207.12s | other issues to be fixed. Um, and yeah, we may have some time for a few questions if people 2207.20s - 2239.27s | have them. Sorry, what was that? Yeah, we, we took a look at them. Um, I would just say any, 2239.27s - 2253.36s | okay, sounds good. Yeah. Any other password managers? Yes. It was just us over the course of 2253.36s - 2258.32s | about a month. And the, the guy to your left helped us a bit with the last coolest 2258.32s - 2272.92s | vulnerability right there. I'm, I'm so sorry. Can you say that again? Oh, just, we wanted to 2272.94s - 2277.40s | know how safe our passwords were. You know, really the detections, I would say are the most 2277.40s - 2282.58s | important part for us. And I would say, practically, we expected this as, we expected this 2282.58s - 2287.08s | as likely possible with, you know, most major password managers or similar attacks. We 2287.08s - 2290.78s | wanted to do it on 1Password because we use 1Password and we wanted to use it on red team 2290.78s - 2307.20s | operations. Um, and then of course get the issues fixed as well. So, uh, we, we took a brief 2307.20s - 2312.10s | look at other password managers, but this was a significant amount of effort for us. And so you 2312.12s - 2317.78s | can imagine that we wouldn't want to do the work, uh, especially on, you know, work time if, 2317.78s


chunk_id: 216_chunk_44
chunk_text:
was just us over the course of 2253.36s - 2258.32s | about a month. And the, the guy to your left helped us a bit with the last coolest 2258.32s - 2272.92s | vulnerability right there. I'm, I'm so sorry. Can you say that again? Oh, just, we wanted to 2272.94s - 2277.40s | know how safe our passwords were. You know, really the detections, I would say are the most 2277.40s - 2282.58s | important part for us. And I would say, practically, we expected this as, we expected this 2282.58s - 2287.08s | as likely possible with, you know, most major password managers or similar attacks. We 2287.08s - 2290.78s | wanted to do it on 1Password because we use 1Password and we wanted to use it on red team 2290.78s - 2307.20s | operations. Um, and then of course get the issues fixed as well. So, uh, we, we took a brief 2307.20s - 2312.10s | look at other password managers, but this was a significant amount of effort for us. And so you 2312.12s - 2317.78s | can imagine that we wouldn't want to do the work, uh, especially on, you know, work time if, 2317.78s - 2328.10s | uh, if it wasn't going to pay out on a red team. All right. Thank you.


chunk_id: 217_chunk_0
chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=5P7KatZBr_I", "title": "DEF CON 32 - Grand Theft Actions Abusing Self Hosted GitHub Runners - Adnan Khan, John Stawinski", "description": "GitHub Actions is quickly becoming the de facto CI/CD provider for open-source projects, startups, and enterprises. At the same time, GitHub\u2019s security model is full of insecure defaults. This makes it easy for their customers to expose themselves to critical attacks from the public internet. The end result? A systemic vulnerability class that won\u2019t go away.\n\nDuring our research, we identified GitHub Actions misconfigurations at scale that would allow threat actors to backdoor major open-source projects. An example of this is our attack on PyTorch, a prominent ML framework used by companies and researchers around the world.\n\nThrough this attack, we could contribute code directly to the main branch of the PyTorch repository, upload malicious releases, backdoor other PyTorch projects, and more. These attacks began by compromising self-hosted runners, which are machines that execute jobs in a GitHub Actions workflow. From there, we leveraged misconfigurations and GitHub \u201cfeatures\u201d to elevate our privileges within GitHub Actions workflows.\n\nOur research campaign included dozens of reports, over $250,000 in bug bounties, and endless war stories. Tune in for a deep dive into the TTPs that allow turning a trivial runner compromise into a full supply chain attack.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 2663, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 1.94s - 4.76s | This text was transcribed using whisper model: large-v2 All right, hello everyone. 4.76s - 7.60s | All right, let's address the burning question in the room. 7.60s - 10.08s | No, we did not get access to the source code 10.08s - 11.48s | for Grand Theft Auto 6, 11.48s - 14.00s | nor do we know if the game will ever be released. 14.00s - 16.96s | Maybe in the following presentation, but not today. 16.96s - 18.60s | This is Grand Theft Actions 18.60s - 21.12s | abusing self-hosted GitHub runners at scale. 22.20s - 23.94s | Hello, everybody. 23.94s - 25.60s | Before we get started, we read online 25.60s - 26.80s | that as long as we have a disclaimer, 26.80s - 29.68s | nobody's allowed to sue us, so here we go. 29.68s - 31.40s | All vulnerabilities mentioned during this talk 31.40s - 32.80s | have been remediated. 32.80s - 35.02s | The views and opinions expressed are solely our own 35.02s - 37.12s | and


chunk_id: 217_chunk_1
chunk_text:
that allow turning a trivial runner compromise into a full supply chain attack.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 2663, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 1.94s - 4.76s | This text was transcribed using whisper model: large-v2 All right, hello everyone. 4.76s - 7.60s | All right, let's address the burning question in the room. 7.60s - 10.08s | No, we did not get access to the source code 10.08s - 11.48s | for Grand Theft Auto 6, 11.48s - 14.00s | nor do we know if the game will ever be released. 14.00s - 16.96s | Maybe in the following presentation, but not today. 16.96s - 18.60s | This is Grand Theft Actions 18.60s - 21.12s | abusing self-hosted GitHub runners at scale. 22.20s - 23.94s | Hello, everybody. 23.94s - 25.60s | Before we get started, we read online 25.60s - 26.80s | that as long as we have a disclaimer, 26.80s - 29.68s | nobody's allowed to sue us, so here we go. 29.68s - 31.40s | All vulnerabilities mentioned during this talk 31.40s - 32.80s | have been remediated. 32.80s - 35.02s | The views and opinions expressed are solely our own 35.02s - 37.12s | and the content is not endorsed by, 37.12s - 40.20s | nor does it represent the views of our employers. 40.20s - 42.36s | All right, whoa, there we go. 42.36s - 44.00s | Adnan, I think we're good to go. 47.48s - 48.92s | All right, if you're wondering about us, 48.92s - 50.68s | so my name is Adnan Khan. 50.68s - 52.96s | For my day job, I work as a security engineer. 52.96s - 54.72s | I'm also a security researcher 54.72s - 56.10s | and bug bounty hunter on the side, 56.10s - 58.56s | and you can find some of my socials below. 58.56s - 59.80s | My name is John Stawinski. 59.80s - 62.24s | I'm a red team security engineer at Praetorian. 62.24s - 65.36s | I also do CICD security research on the side. 65.36s - 67.08s | In the last year, I've watched Avatar, 67.08s - 69.92s | The Last Airbender, three times, the animated version, 69.92s - 75.32s | and in a past life, I was a collegiate wrestler. 75.32s - 76.96s | All right, so we're going to be talking a lot 76.96s - 78.32s | about self-hosted runners


chunk_id: 217_chunk_2
chunk_text:
the content is not endorsed by, 37.12s - 40.20s | nor does it represent the views of our employers. 40.20s - 42.36s | All right, whoa, there we go. 42.36s - 44.00s | Adnan, I think we're good to go. 47.48s - 48.92s | All right, if you're wondering about us, 48.92s - 50.68s | so my name is Adnan Khan. 50.68s - 52.96s | For my day job, I work as a security engineer. 52.96s - 54.72s | I'm also a security researcher 54.72s - 56.10s | and bug bounty hunter on the side, 56.10s - 58.56s | and you can find some of my socials below. 58.56s - 59.80s | My name is John Stawinski. 59.80s - 62.24s | I'm a red team security engineer at Praetorian. 62.24s - 65.36s | I also do CICD security research on the side. 65.36s - 67.08s | In the last year, I've watched Avatar, 67.08s - 69.92s | The Last Airbender, three times, the animated version, 69.92s - 75.32s | and in a past life, I was a collegiate wrestler. 75.32s - 76.96s | All right, so we're going to be talking a lot 76.96s - 78.32s | about self-hosted runners today, 78.32s - 80.08s | so I want to make sure everyone has a background 80.08s - 82.28s | and understanding of how GitHub runners work. 82.28s - 84.48s | On one side, you have GitHub-hosted runners. 84.48s - 87.24s | These are built by GitHub, updated frequently, 87.24s - 89.04s | and as of writing, cover a wide variety 89.04s - 91.06s | of operating systems and architectures. 91.06s - 92.98s | The key thing with GitHub-hosted runners 92.98s - 96.34s | is that they are ephemeral, meaning they are torn down 96.34s - 99.18s | at the conclusion of each workflow job. 99.18s - 101.74s | On the other hand, you have self-hosted runners. 101.74s - 105.34s | These are managed and secured entirely by the end user, 105.34s - 107.22s | and it's not surprising here 107.22s - 109.26s | that the path of least resistance 109.26s - 112.30s | when configuring one of these self-hosted runners 112.30s - 114.50s | is a non-ephemeral self-hosted runner 114.50s - 116.58s | that is actually the least secure. 116.58s - 119.06s | Both GitHub-hosted and self-hosted runners 119.06s - 121.34s | work to service GitHub Actions workflows 121.34s -


chunk_id: 217_chunk_3
chunk_text:
today, 78.32s - 80.08s | so I want to make sure everyone has a background 80.08s - 82.28s | and understanding of how GitHub runners work. 82.28s - 84.48s | On one side, you have GitHub-hosted runners. 84.48s - 87.24s | These are built by GitHub, updated frequently, 87.24s - 89.04s | and as of writing, cover a wide variety 89.04s - 91.06s | of operating systems and architectures. 91.06s - 92.98s | The key thing with GitHub-hosted runners 92.98s - 96.34s | is that they are ephemeral, meaning they are torn down 96.34s - 99.18s | at the conclusion of each workflow job. 99.18s - 101.74s | On the other hand, you have self-hosted runners. 101.74s - 105.34s | These are managed and secured entirely by the end user, 105.34s - 107.22s | and it's not surprising here 107.22s - 109.26s | that the path of least resistance 109.26s - 112.30s | when configuring one of these self-hosted runners 112.30s - 114.50s | is a non-ephemeral self-hosted runner 114.50s - 116.58s | that is actually the least secure. 116.58s - 119.06s | Both GitHub-hosted and self-hosted runners 119.06s - 121.34s | work to service GitHub Actions workflows 121.34s - 123.94s | that are triggered from repositories on GitHub. 124.86s - 126.42s | So why are we here today? 126.42s - 129.18s | Well, all the organizations and projects 129.18s - 130.54s | that you see on the screen here 130.54s - 133.10s | use or at one point use self-hosted runners 133.10s - 135.34s | on their public GitHub repositories. 135.34s - 138.04s | Not only that, but they use them in an insecure manner, 138.04s - 139.64s | and over the last 12 months, 139.64s - 141.62s | we were able to identify vulnerabilities 141.62s - 143.66s | in these organizations and many more 143.66s - 144.94s | that we can't talk about. 144.94s - 147.46s | Some of these cases could have led to widespread, 147.46s - 149.24s | critical supply chain attacks. 150.82s - 153.22s | You may be wondering if we just showed that last slide 153.22s - 154.82s | to flex our cool bones at DEF CON, 154.82s - 157.58s | and I just want to go on record saying that, 157.58s - 159.86s | yeah, pretty much. 159.86s - 162.40s | But it also raises a bigger question. 162.40s - 165.22s | Why


chunk_id: 217_chunk_4
chunk_text:
123.94s | that are triggered from repositories on GitHub. 124.86s - 126.42s | So why are we here today? 126.42s - 129.18s | Well, all the organizations and projects 129.18s - 130.54s | that you see on the screen here 130.54s - 133.10s | use or at one point use self-hosted runners 133.10s - 135.34s | on their public GitHub repositories. 135.34s - 138.04s | Not only that, but they use them in an insecure manner, 138.04s - 139.64s | and over the last 12 months, 139.64s - 141.62s | we were able to identify vulnerabilities 141.62s - 143.66s | in these organizations and many more 143.66s - 144.94s | that we can't talk about. 144.94s - 147.46s | Some of these cases could have led to widespread, 147.46s - 149.24s | critical supply chain attacks. 150.82s - 153.22s | You may be wondering if we just showed that last slide 153.22s - 154.82s | to flex our cool bones at DEF CON, 154.82s - 157.58s | and I just want to go on record saying that, 157.58s - 159.86s | yeah, pretty much. 159.86s - 162.40s | But it also raises a bigger question. 162.40s - 165.22s | Why were we in a position to execute supply chain attacks 165.22s - 166.66s | on all of these companies? 166.66s - 167.86s | And the reason behind that 167.86s - 171.02s | is that GitHub Actions provides this broad attack surface 171.02s - 173.58s | that exposes these organizations to compromise, 173.58s - 176.46s | especially when they use self-hosted runners. 177.70s - 179.78s | All right, so how do we get started 179.78s - 181.42s | with our vulnerability research campaign? 181.42s - 185.14s | Well, it all started all the way back in August of 2022 185.14s - 188.06s | when I was working on a red team with my former employer. 188.06s - 189.66s | I was fairly new to red teaming at the time, 189.66s - 190.50s | and I tripped a canary 190.50s - 191.94s | and got us kicked out of the client network. 191.94s - 193.76s | Maybe that's happened to some of y'all. 193.76s - 195.10s | But while trying to get back in, 195.10s - 196.24s | we did some social engineering, 196.24s - 198.74s | and I found that we could use a GitHub access token 198.74s - 200.66s |


chunk_id: 217_chunk_5
chunk_text:
were we in a position to execute supply chain attacks 165.22s - 166.66s | on all of these companies? 166.66s - 167.86s | And the reason behind that 167.86s - 171.02s | is that GitHub Actions provides this broad attack surface 171.02s - 173.58s | that exposes these organizations to compromise, 173.58s - 176.46s | especially when they use self-hosted runners. 177.70s - 179.78s | All right, so how do we get started 179.78s - 181.42s | with our vulnerability research campaign? 181.42s - 185.14s | Well, it all started all the way back in August of 2022 185.14s - 188.06s | when I was working on a red team with my former employer. 188.06s - 189.66s | I was fairly new to red teaming at the time, 189.66s - 190.50s | and I tripped a canary 190.50s - 191.94s | and got us kicked out of the client network. 191.94s - 193.76s | Maybe that's happened to some of y'all. 193.76s - 195.10s | But while trying to get back in, 195.10s - 196.24s | we did some social engineering, 196.24s - 198.74s | and I found that we could use a GitHub access token 198.74s - 200.66s | that we obtained to execute a workflow 200.66s - 203.02s | and persist on a non-ephemeral self-hosted runner 203.02s - 204.22s | inside that client's network, 204.22s - 206.94s | and that got us back in and we reached objective. 206.94s - 209.02s | That led to a talk at ShmooCon 209.02s - 210.30s | called Phantom of the Pipeline 210.30s - 212.10s | that was given in January of 2023 212.10s - 214.70s | and the release of the original Gato tool. 215.90s - 218.26s | Around the time that Anand, Mason Davis, 218.26s - 220.42s | and Matt Chkosky were developing Gato, 220.42s - 221.50s | I joined them on the red team, 221.50s - 223.36s | and Gato was my entry to CICD Security. 223.36s - 225.02s | So I started using Gato 225.02s - 227.02s | to identify GitHub Actions vulnerabilities 227.02s - 228.46s | during red team engagements, 228.46s - 231.86s | and then we started diving deeper into GitHub Actions abuse, 231.86s - 234.06s | especially around post-exploitation. 234.06s - 235.32s | And as we were doing our research, 235.32s - 236.92s | we were trying to brainstorm ways 237.76s - 238.60s | to


chunk_id: 217_chunk_6
chunk_text:
that we obtained to execute a workflow 200.66s - 203.02s | and persist on a non-ephemeral self-hosted runner 203.02s - 204.22s | inside that client's network, 204.22s - 206.94s | and that got us back in and we reached objective. 206.94s - 209.02s | That led to a talk at ShmooCon 209.02s - 210.30s | called Phantom of the Pipeline 210.30s - 212.10s | that was given in January of 2023 212.10s - 214.70s | and the release of the original Gato tool. 215.90s - 218.26s | Around the time that Anand, Mason Davis, 218.26s - 220.42s | and Matt Chkosky were developing Gato, 220.42s - 221.50s | I joined them on the red team, 221.50s - 223.36s | and Gato was my entry to CICD Security. 223.36s - 225.02s | So I started using Gato 225.02s - 227.02s | to identify GitHub Actions vulnerabilities 227.02s - 228.46s | during red team engagements, 228.46s - 231.86s | and then we started diving deeper into GitHub Actions abuse, 231.86s - 234.06s | especially around post-exploitation. 234.06s - 235.32s | And as we were doing our research, 235.32s - 236.92s | we were trying to brainstorm ways 237.76s - 238.60s | to blow this up and bring it 238.60s - 241.20s | from internal red team engagements to everybody, 241.20s - 242.52s | and Anand had an idea 242.52s - 244.64s | that really took it to the next level. 244.64s - 246.12s | So one of the little asterisks 246.12s - 247.84s | at the end of the ShmooCon presentation 247.84s - 250.72s | was that if you fixed a typo on a public repository 250.72s - 253.16s | that was using a non-ephemeral self-hosted runner, 253.16s - 254.48s | you could just make a pull request 254.48s - 257.12s | and modify the workflow file and get persistence. 257.12s - 258.92s | I demonstrated that vulnerability 258.92s - 261.82s | against GitHub Actions runner images repository, 261.82s - 264.36s | which was accepted as a critical vulnerability. 265.36s - 268.40s | After Anand hacked GitHub Actions runner images, 268.40s - 271.08s | we started looking at all public repositories 271.08s - 273.28s | and realized that these misconfigurations were everywhere. 273.28s - 274.80s | So we decided to team up. 274.80s - 276.00s | Our first joint operation 276.00s - 277.88s | was breaching Microsoft's perimeter 277.88s - 280.24s |


chunk_id: 217_chunk_7
chunk_text:
blow this up and bring it 238.60s - 241.20s | from internal red team engagements to everybody, 241.20s - 242.52s | and Anand had an idea 242.52s - 244.64s | that really took it to the next level. 244.64s - 246.12s | So one of the little asterisks 246.12s - 247.84s | at the end of the ShmooCon presentation 247.84s - 250.72s | was that if you fixed a typo on a public repository 250.72s - 253.16s | that was using a non-ephemeral self-hosted runner, 253.16s - 254.48s | you could just make a pull request 254.48s - 257.12s | and modify the workflow file and get persistence. 257.12s - 258.92s | I demonstrated that vulnerability 258.92s - 261.82s | against GitHub Actions runner images repository, 261.82s - 264.36s | which was accepted as a critical vulnerability. 265.36s - 268.40s | After Anand hacked GitHub Actions runner images, 268.40s - 271.08s | we started looking at all public repositories 271.08s - 273.28s | and realized that these misconfigurations were everywhere. 273.28s - 274.80s | So we decided to team up. 274.80s - 276.00s | Our first joint operation 276.00s - 277.88s | was breaching Microsoft's perimeter 277.88s - 280.24s | by getting code execution on a domain-joined machine 280.24s - 282.04s | through Microsoft DeepSpeed, 282.04s - 285.52s | and then we launched a series of attacks, 285.52s - 288.44s | the next one of which we will cover today. 288.44s - 290.16s | All right, so at a high level, 290.16s - 291.60s | there's three steps to discovering 291.60s - 293.56s | this vulnerability at scale. 293.56s - 296.36s | The first step is to search for candidate repositories 296.36s - 298.32s | that might be using a self-hosted runner. 298.32s - 301.56s | To do this, we used a combination of GitHub CodeSearch 301.56s - 304.28s | and Sourcegraph CodeSearch dorks. 304.28s - 305.56s | Both have pros and cons, 305.56s - 309.00s | so we used both and deduplicated the results 309.00s - 311.10s | to get a nice list of candidate repositories 311.10s - 313.12s | for further analysis. 313.12s - 316.24s | To do this, use the tool that's open source right now 316.24s - 317.50s | called GatoX. 317.50s - 320.32s | It uses automated workflow run log analysis 320.32s - 321.88s | to determine if there's a non-ephemeral 321.88s - 323.48s |


chunk_id: 217_chunk_8
chunk_text:
by getting code execution on a domain-joined machine 280.24s - 282.04s | through Microsoft DeepSpeed, 282.04s - 285.52s | and then we launched a series of attacks, 285.52s - 288.44s | the next one of which we will cover today. 288.44s - 290.16s | All right, so at a high level, 290.16s - 291.60s | there's three steps to discovering 291.60s - 293.56s | this vulnerability at scale. 293.56s - 296.36s | The first step is to search for candidate repositories 296.36s - 298.32s | that might be using a self-hosted runner. 298.32s - 301.56s | To do this, we used a combination of GitHub CodeSearch 301.56s - 304.28s | and Sourcegraph CodeSearch dorks. 304.28s - 305.56s | Both have pros and cons, 305.56s - 309.00s | so we used both and deduplicated the results 309.00s - 311.10s | to get a nice list of candidate repositories 311.10s - 313.12s | for further analysis. 313.12s - 316.24s | To do this, use the tool that's open source right now 316.24s - 317.50s | called GatoX. 317.50s - 320.32s | It uses automated workflow run log analysis 320.32s - 321.88s | to determine if there's a non-ephemeral 321.88s - 323.48s | self-hosted runner in use. 324.28s - 326.58s | Once you have the results, you can plan your next step, 326.58s - 328.76s | which is essentially, can you hack it, 328.76s - 330.82s | and can you hack it for some good impact? 332.24s - 335.36s | Self-hosted runner takeover is a special case 335.36s - 337.70s | of public Poison Pipeline execution, 337.70s - 340.24s | so Poison Pipeline execution at a high level 340.24s - 342.52s | is any time you can get your own code 342.52s - 345.04s | executed by a GitHub Actions workflow, 345.04s - 347.24s | you can conduct a Poison Pipeline attack. 347.24s - 348.68s | In the case of self-hosted runner, 348.68s - 350.24s | we'll use this to deploy persistence 350.24s - 352.56s | on the self-hosted runner via a pull request, 352.56s - 354.96s | and that opens up a ton of lateral movement 354.96s - 356.28s | and privilege escalation paths. 356.28s - 358.80s | So the first two aspects of this diagram 358.80s - 360.12s | are pretty straightforward. 360.12s - 363.28s | This last section, the GitHub Actions post-exploitation, 363.28s - 365.24s | is where we focus


chunk_id: 217_chunk_9
chunk_text:
self-hosted runner in use. 324.28s - 326.58s | Once you have the results, you can plan your next step, 326.58s - 328.76s | which is essentially, can you hack it, 328.76s - 330.82s | and can you hack it for some good impact? 332.24s - 335.36s | Self-hosted runner takeover is a special case 335.36s - 337.70s | of public Poison Pipeline execution, 337.70s - 340.24s | so Poison Pipeline execution at a high level 340.24s - 342.52s | is any time you can get your own code 342.52s - 345.04s | executed by a GitHub Actions workflow, 345.04s - 347.24s | you can conduct a Poison Pipeline attack. 347.24s - 348.68s | In the case of self-hosted runner, 348.68s - 350.24s | we'll use this to deploy persistence 350.24s - 352.56s | on the self-hosted runner via a pull request, 352.56s - 354.96s | and that opens up a ton of lateral movement 354.96s - 356.28s | and privilege escalation paths. 356.28s - 358.80s | So the first two aspects of this diagram 358.80s - 360.12s | are pretty straightforward. 360.12s - 363.28s | This last section, the GitHub Actions post-exploitation, 363.28s - 365.24s | is where we focus most of our research. 366.16s - 368.80s | All right, so it's a tale as old as Unix time itself. 368.80s - 372.24s | Misconfigurations are amplified by insecure defaults. 372.24s - 375.44s | So the default setting for a pull request approval 375.44s - 377.36s | on the pull request trigger is that 377.36s - 380.56s | it only requires approval for first time contributors. 380.56s - 382.28s | So if someone has a pull request, 383.00s - 385.04s | no matter how trivial, merged into the default branch, 385.04s - 387.92s | they won't need approval for workload executions 387.92s - 389.76s | on their pull requests. 389.76s - 394.84s | This is that bad insecure default. 394.84s - 397.56s | Today, we're gonna walk through playing with fire, 397.56s - 399.92s | which is what we've called our supply chain attack 399.92s - 401.64s | on the PyTorch repository, 401.64s - 404.64s | which began by compromising self-hosted runners. 404.64s - 406.36s | This was one of our more complex 406.36s - 408.72s | and more intricate attacks, 408.72s - 410.32s | so we're gonna use it as a case study 410.32s - 412.76s | to teach


chunk_id: 217_chunk_10
chunk_text:
most of our research. 366.16s - 368.80s | All right, so it's a tale as old as Unix time itself. 368.80s - 372.24s | Misconfigurations are amplified by insecure defaults. 372.24s - 375.44s | So the default setting for a pull request approval 375.44s - 377.36s | on the pull request trigger is that 377.36s - 380.56s | it only requires approval for first time contributors. 380.56s - 382.28s | So if someone has a pull request, 383.00s - 385.04s | no matter how trivial, merged into the default branch, 385.04s - 387.92s | they won't need approval for workload executions 387.92s - 389.76s | on their pull requests. 389.76s - 394.84s | This is that bad insecure default. 394.84s - 397.56s | Today, we're gonna walk through playing with fire, 397.56s - 399.92s | which is what we've called our supply chain attack 399.92s - 401.64s | on the PyTorch repository, 401.64s - 404.64s | which began by compromising self-hosted runners. 404.64s - 406.36s | This was one of our more complex 406.36s - 408.72s | and more intricate attacks, 408.72s - 410.32s | so we're gonna use it as a case study 410.32s - 412.76s | to teach technical TTPs of GitHub Actions 412.76s - 415.08s | exploitation and post-exploitation as we go. 416.00s - 417.88s | All right, so what is PyTorch? 417.88s - 419.32s | I'm sure a lot of you already know what it is, 419.32s - 420.76s | but it's a machine learning framer 420.76s - 423.12s | originally developed by Meta that's been open-sourced, 423.12s - 424.88s | and it's used by companies around the world 424.88s - 427.64s | like Google, Lockheed Martin, OpenAI, and more. 427.64s - 429.44s | If someone were to backdoor PyTorch, 429.44s - 431.32s | this could allow compromising developers 431.32s - 433.08s | and organizations in that very much 433.08s - 435.06s | red-hot AIML space right now. 436.56s - 439.68s | In all seriousness, probably the hardest part 439.68s - 442.40s | of compromising PyTorch was searching 442.92s - 443.74s | through their workflows. 443.74s - 446.16s | They had something like over 90 workflows, 446.16s - 447.64s | over 15 GitHub secrets, 447.64s - 449.52s | and more than five self-hosted runners 449.52s - 451.12s | on their public repository, 451.12s - 453.56s | and I vividly remember sitting at my desk. 453.56s -


chunk_id: 217_chunk_11
chunk_text:
technical TTPs of GitHub Actions 412.76s - 415.08s | exploitation and post-exploitation as we go. 416.00s - 417.88s | All right, so what is PyTorch? 417.88s - 419.32s | I'm sure a lot of you already know what it is, 419.32s - 420.76s | but it's a machine learning framer 420.76s - 423.12s | originally developed by Meta that's been open-sourced, 423.12s - 424.88s | and it's used by companies around the world 424.88s - 427.64s | like Google, Lockheed Martin, OpenAI, and more. 427.64s - 429.44s | If someone were to backdoor PyTorch, 429.44s - 431.32s | this could allow compromising developers 431.32s - 433.08s | and organizations in that very much 433.08s - 435.06s | red-hot AIML space right now. 436.56s - 439.68s | In all seriousness, probably the hardest part 439.68s - 442.40s | of compromising PyTorch was searching 442.92s - 443.74s | through their workflows. 443.74s - 446.16s | They had something like over 90 workflows, 446.16s - 447.64s | over 15 GitHub secrets, 447.64s - 449.52s | and more than five self-hosted runners 449.52s - 451.12s | on their public repository, 451.12s - 453.56s | and I vividly remember sitting at my desk. 453.56s - 455.72s | I had like three different notebooks out, 455.72s - 457.76s | and I was manually, pen and paper, 457.76s - 460.56s | mapping out the data flow through all these workflows, 460.56s - 463.44s | a lot of which were nested in preparing for this attack 463.44s - 465.52s | so that we could try to figure out the potential impact 465.52s - 466.88s | before we actually started. 468.08s - 469.88s | When we confirm that a repository 469.88s - 471.74s | has a self-hosted runner, 471.74s - 473.02s | we look at the workflow logs 473.02s - 474.50s | like Adnan talked about earlier. 474.50s - 477.06s | So here's an example of the PyTorch repository, 477.06s - 478.42s | and in this screenshot, 478.42s - 479.66s | you can see in the workflow log, 479.66s - 481.02s | it has the runner name, 481.02s - 481.88s | the runner group name, 481.88s - 482.72s | and the machine name. 482.72s - 484.78s | So we see that this is a Jenkins runner. 484.78s - 485.82s | It's a self-hosted runner 485.82s - 489.20s | attached to their public repository in the default group, 489.20s -


chunk_id: 217_chunk_12
chunk_text:
455.72s | I had like three different notebooks out, 455.72s - 457.76s | and I was manually, pen and paper, 457.76s - 460.56s | mapping out the data flow through all these workflows, 460.56s - 463.44s | a lot of which were nested in preparing for this attack 463.44s - 465.52s | so that we could try to figure out the potential impact 465.52s - 466.88s | before we actually started. 468.08s - 469.88s | When we confirm that a repository 469.88s - 471.74s | has a self-hosted runner, 471.74s - 473.02s | we look at the workflow logs 473.02s - 474.50s | like Adnan talked about earlier. 474.50s - 477.06s | So here's an example of the PyTorch repository, 477.06s - 478.42s | and in this screenshot, 478.42s - 479.66s | you can see in the workflow log, 479.66s - 481.02s | it has the runner name, 481.02s - 481.88s | the runner group name, 481.88s - 482.72s | and the machine name. 482.72s - 484.78s | So we see that this is a Jenkins runner. 484.78s - 485.82s | It's a self-hosted runner 485.82s - 489.20s | attached to their public repository in the default group, 489.20s - 490.46s | and this is one of the runners 490.46s - 491.96s | that we ended up compromising. 492.94s - 495.46s | All right, remember that default approval requirement? 495.46s - 498.02s | Well, for certain repositories that are very busy, 498.02s - 499.62s | you can often determine that 499.62s - 501.10s | just using contextual analysis, 501.10s - 501.94s | and let me tell you, 501.94s - 503.54s | PyTorch was very busy. 503.54s - 505.54s | So what you look for is a pull request 505.54s - 507.02s | submitted by a previous contributor 507.02s - 508.70s | that's from a fork pull request. 508.70s - 510.50s | You check that the workflow run 510.50s - 512.30s | associated with it was not approved, 512.30s - 515.88s | and that it ran on the pull request trigger. 515.88s - 517.10s | When all these things come together, 517.10s - 519.50s | it's very likely that the default approval requirements 519.50s - 520.34s | are in place. 521.66s - 524.62s | GitHub warns against attaching self-hosted runners 524.62s - 526.70s | to public repositories for obvious reasons, 526.70s - 528.98s | but they don't make their documentation 528.98s


chunk_id: 217_chunk_13
chunk_text:
490.46s | and this is one of the runners 490.46s - 491.96s | that we ended up compromising. 492.94s - 495.46s | All right, remember that default approval requirement? 495.46s - 498.02s | Well, for certain repositories that are very busy, 498.02s - 499.62s | you can often determine that 499.62s - 501.10s | just using contextual analysis, 501.10s - 501.94s | and let me tell you, 501.94s - 503.54s | PyTorch was very busy. 503.54s - 505.54s | So what you look for is a pull request 505.54s - 507.02s | submitted by a previous contributor 507.02s - 508.70s | that's from a fork pull request. 508.70s - 510.50s | You check that the workflow run 510.50s - 512.30s | associated with it was not approved, 512.30s - 515.88s | and that it ran on the pull request trigger. 515.88s - 517.10s | When all these things come together, 517.10s - 519.50s | it's very likely that the default approval requirements 519.50s - 520.34s | are in place. 521.66s - 524.62s | GitHub warns against attaching self-hosted runners 524.62s - 526.70s | to public repositories for obvious reasons, 526.70s - 528.98s | but they don't make their documentation 528.98s - 529.82s | and their warnings obvious, 529.82s - 530.98s | and you kind of have to go out of your way. 531.66s - 533.58s | Part of the reason we think that so many repos 533.58s - 535.14s | publicly use self-hosted runners 535.14s - 537.82s | is because they don't end up seeing this documentation. 537.82s - 538.78s | In this next slide, 538.78s - 540.34s | we'll just walk through what it looks like 540.34s - 542.06s | to register a self-hosted runner 542.06s - 545.46s | and see what documentation you come across as a developer. 546.42s - 548.06s | So in my GitHub repository, 548.06s - 549.48s | I'm going to actions panel. 549.48s - 551.14s | I'm trying to register a self-hosted runner, 551.14s - 553.34s | but first I'm gonna look through their documentation 553.34s - 554.58s | because I'm a good developer, 554.58s - 556.08s | and I click on these links. 556.08s - 557.58s | I don't see anything security related, 557.58s - 558.78s | so I click on this other link, 558.78s - 560.90s | and we get to a lot of GitHub 561.70s - 562.86s | self-hosted runner documentation.


chunk_id: 217_chunk_14
chunk_text:
- 529.82s | and their warnings obvious, 529.82s - 530.98s | and you kind of have to go out of your way. 531.66s - 533.58s | Part of the reason we think that so many repos 533.58s - 535.14s | publicly use self-hosted runners 535.14s - 537.82s | is because they don't end up seeing this documentation. 537.82s - 538.78s | In this next slide, 538.78s - 540.34s | we'll just walk through what it looks like 540.34s - 542.06s | to register a self-hosted runner 542.06s - 545.46s | and see what documentation you come across as a developer. 546.42s - 548.06s | So in my GitHub repository, 548.06s - 549.48s | I'm going to actions panel. 549.48s - 551.14s | I'm trying to register a self-hosted runner, 551.14s - 553.34s | but first I'm gonna look through their documentation 553.34s - 554.58s | because I'm a good developer, 554.58s - 556.08s | and I click on these links. 556.08s - 557.58s | I don't see anything security related, 557.58s - 558.78s | so I click on this other link, 558.78s - 560.90s | and we get to a lot of GitHub 561.70s - 562.86s | self-hosted runner documentation. 562.86s - 566.46s | We learn about auto-scaling, usage limits. 566.46s - 568.22s | I think it lists architecture stuff, 568.22s - 570.70s | and then every API that the runner's going to use 570.70s - 571.98s | in its lifetime, 571.98s - 574.62s | and then way down here at the bottom, 574.62s - 576.42s | it talks about self-hosted runner security, 576.42s - 579.20s | and so GitHub does clearly say 579.20s - 581.46s | they don't recommend using self-hosted GitHub runners 581.46s - 582.86s | on public repositories, 582.86s - 584.90s | and they have other articles out there too 584.90s - 586.98s | elaborating on this, 586.98s - 589.74s | but the point we want to drive is 589.74s - 590.86s | it's not obvious. 590.86s - 591.70s | So chances are, 591.70s - 593.30s | if you're just registering a self-hosted runner 593.30s - 594.50s | to a public repo, 594.50s - 597.58s | you're not going to come across this documentation. 597.58s - 598.74s | All right, the first step 598.74s - 600.90s | was to infiltrate the contributor list. 600.90s - 602.78s | This is what's important to take advantage 602.78s - 604.54s


chunk_id: 217_chunk_15
chunk_text:
562.86s - 566.46s | We learn about auto-scaling, usage limits. 566.46s - 568.22s | I think it lists architecture stuff, 568.22s - 570.70s | and then every API that the runner's going to use 570.70s - 571.98s | in its lifetime, 571.98s - 574.62s | and then way down here at the bottom, 574.62s - 576.42s | it talks about self-hosted runner security, 576.42s - 579.20s | and so GitHub does clearly say 579.20s - 581.46s | they don't recommend using self-hosted GitHub runners 581.46s - 582.86s | on public repositories, 582.86s - 584.90s | and they have other articles out there too 584.90s - 586.98s | elaborating on this, 586.98s - 589.74s | but the point we want to drive is 589.74s - 590.86s | it's not obvious. 590.86s - 591.70s | So chances are, 591.70s - 593.30s | if you're just registering a self-hosted runner 593.30s - 594.50s | to a public repo, 594.50s - 597.58s | you're not going to come across this documentation. 597.58s - 598.74s | All right, the first step 598.74s - 600.90s | was to infiltrate the contributor list. 600.90s - 602.78s | This is what's important to take advantage 602.78s - 604.54s | of those default approval settings. 605.46s - 606.98s | To infiltrate the contributor list, 606.98s - 608.24s | if you've been following our blogs at all, 608.24s - 609.82s | we always call the grammar police. 609.82s - 612.36s | So the grammar police is me and Adnan. 612.36s - 613.32s | We show up, 613.32s - 615.18s | and we run all of their markdown files 615.18s - 616.34s | through Grammarly. 616.34s - 618.22s | Again, fixing a typo here 618.22s - 619.70s | and finding a grammatical error 619.70s - 620.70s | doesn't matter at all. 620.70s - 622.72s | To become a contributor, 622.72s - 624.30s | we also could make a legitimate, 624.30s - 625.82s | substantial code contribution 625.82s - 627.62s | to the PyTorch repository, 627.62s - 630.00s | but that's a lot of work. 630.00s - 632.10s | So what we do instead is we notice 632.10s - 634.26s | resolve was in the past tense, 634.26s - 635.66s | and it should have been in the present tense. 635.66s - 637.46s | So we fixed that for them, 637.46s - 639.58s | and we submitted a pull request, 639.58s -


chunk_id: 217_chunk_16
chunk_text:
| of those default approval settings. 605.46s - 606.98s | To infiltrate the contributor list, 606.98s - 608.24s | if you've been following our blogs at all, 608.24s - 609.82s | we always call the grammar police. 609.82s - 612.36s | So the grammar police is me and Adnan. 612.36s - 613.32s | We show up, 613.32s - 615.18s | and we run all of their markdown files 615.18s - 616.34s | through Grammarly. 616.34s - 618.22s | Again, fixing a typo here 618.22s - 619.70s | and finding a grammatical error 619.70s - 620.70s | doesn't matter at all. 620.70s - 622.72s | To become a contributor, 622.72s - 624.30s | we also could make a legitimate, 624.30s - 625.82s | substantial code contribution 625.82s - 627.62s | to the PyTorch repository, 627.62s - 630.00s | but that's a lot of work. 630.00s - 632.10s | So what we do instead is we notice 632.10s - 634.26s | resolve was in the past tense, 634.26s - 635.66s | and it should have been in the present tense. 635.66s - 637.46s | So we fixed that for them, 637.46s - 639.58s | and we submitted a pull request, 639.58s - 640.46s | and a few days later, 640.46s - 642.42s | it got merged by the maintainers, 642.42s - 644.52s | and then we were now officially contributors, 644.52s - 646.26s | which is not a security boundary, right? 646.26s - 647.80s | And this is the same process we did 647.80s - 649.40s | with all of our exploits. 649.40s - 651.32s | All right, so this is where the hacking starts. 651.32s - 653.16s | So the step, phase two, 653.16s - 654.92s | was to install command and control 654.92s - 657.08s | on those non-ephemeral self-hosted runners 657.08s - 658.40s | attached to PyTorch. 658.40s - 659.92s | To do this, we leverage something 659.92s - 662.04s | we like to call runner-on-runner. 662.04s - 663.10s | In a nutshell, what that is, 663.10s - 665.40s | is we install another self-hosted runner 665.40s - 667.00s | on their self-hosted runner. 667.00s - 669.52s | This has the benefit of sending all traffic 669.52s - 672.00s | to the same domains and IP addresses 672.00s - 674.04s | that the original self-hosted runner uses. 674.04s - 676.58s | So if there is any EDR or monitoring


chunk_id: 217_chunk_17
chunk_text:
640.46s | and a few days later, 640.46s - 642.42s | it got merged by the maintainers, 642.42s - 644.52s | and then we were now officially contributors, 644.52s - 646.26s | which is not a security boundary, right? 646.26s - 647.80s | And this is the same process we did 647.80s - 649.40s | with all of our exploits. 649.40s - 651.32s | All right, so this is where the hacking starts. 651.32s - 653.16s | So the step, phase two, 653.16s - 654.92s | was to install command and control 654.92s - 657.08s | on those non-ephemeral self-hosted runners 657.08s - 658.40s | attached to PyTorch. 658.40s - 659.92s | To do this, we leverage something 659.92s - 662.04s | we like to call runner-on-runner. 662.04s - 663.10s | In a nutshell, what that is, 663.10s - 665.40s | is we install another self-hosted runner 665.40s - 667.00s | on their self-hosted runner. 667.00s - 669.52s | This has the benefit of sending all traffic 669.52s - 672.00s | to the same domains and IP addresses 672.00s - 674.04s | that the original self-hosted runner uses. 674.04s - 676.58s | So if there is any EDR or monitoring software 676.58s - 677.82s | on that runner, it's very likely 677.82s - 679.72s | that we'll fly right under the radar. 679.72s - 681.12s | Oh yeah, so remember that default? 681.12s - 683.70s | This is where those defaults lead to an actual compromise. 683.70s - 687.02s | To do this, we just modified the workflow in our fork 687.02s - 689.40s | and had it just install our C2. 690.54s - 693.70s | The second runner is attached to our private C2 repository, 693.70s - 695.80s | so we can task it to execute commands. 695.80s - 697.86s | It's essentially a web shell. 697.86s - 699.10s | Using this web shell, 699.10s - 701.54s | we're able to look around the file system 701.54s - 703.16s | and learn about the host, 703.16s - 705.98s | and all of this is in preparation for phase three. 707.10s - 709.18s | Phase three is the most important phase 709.18s - 711.42s | of all of our CICD compromise. 711.42s - 714.26s | So the great secret heist, 714.26s - 716.06s | self-hosted runner post-exploitation, 716.06s - 718.66s | is how you go from trivial RCE 718.66s -


chunk_id: 217_chunk_18
chunk_text:
software 676.58s - 677.82s | on that runner, it's very likely 677.82s - 679.72s | that we'll fly right under the radar. 679.72s - 681.12s | Oh yeah, so remember that default? 681.12s - 683.70s | This is where those defaults lead to an actual compromise. 683.70s - 687.02s | To do this, we just modified the workflow in our fork 687.02s - 689.40s | and had it just install our C2. 690.54s - 693.70s | The second runner is attached to our private C2 repository, 693.70s - 695.80s | so we can task it to execute commands. 695.80s - 697.86s | It's essentially a web shell. 697.86s - 699.10s | Using this web shell, 699.10s - 701.54s | we're able to look around the file system 701.54s - 703.16s | and learn about the host, 703.16s - 705.98s | and all of this is in preparation for phase three. 707.10s - 709.18s | Phase three is the most important phase 709.18s - 711.42s | of all of our CICD compromise. 711.42s - 714.26s | So the great secret heist, 714.26s - 716.06s | self-hosted runner post-exploitation, 716.06s - 718.66s | is how you go from trivial RCE 718.66s - 720.02s | to complete supply chain attack. 720.02s - 721.42s | And when we started our research, 721.42s - 723.38s | we submitted some reports and we saw some reports 723.38s - 725.54s | where researchers would get code execution 725.54s - 727.66s | on self-hosted runners, and then they'd say, 727.66s - 729.46s | hey, and by the way, you can probably do 729.46s - 732.02s | all these theoretical post-exploitation stuff. 732.02s - 734.86s | And orgs just didn't care at all. 734.86s - 737.06s | They should care if you're executing code on the runners, 737.06s - 739.86s | but it's kind of what they're supposed to do anyway, 739.86s - 741.98s | and so they didn't realize the actual impact 741.98s - 744.58s | of these attacks unless you go and demonstrate it. 744.58s - 747.22s | So that is what we did. 747.22s - 749.34s | All right, so to understand how we were able 749.34s - 750.58s | to achieve some of this impact, 750.58s - 752.02s | I want to make sure everyone understands 752.02s - 753.46s | the magical GitHub token. 753.46s - 755.50s | So this is a token that is


chunk_id: 217_chunk_19
chunk_text:
720.02s | to complete supply chain attack. 720.02s - 721.42s | And when we started our research, 721.42s - 723.38s | we submitted some reports and we saw some reports 723.38s - 725.54s | where researchers would get code execution 725.54s - 727.66s | on self-hosted runners, and then they'd say, 727.66s - 729.46s | hey, and by the way, you can probably do 729.46s - 732.02s | all these theoretical post-exploitation stuff. 732.02s - 734.86s | And orgs just didn't care at all. 734.86s - 737.06s | They should care if you're executing code on the runners, 737.06s - 739.86s | but it's kind of what they're supposed to do anyway, 739.86s - 741.98s | and so they didn't realize the actual impact 741.98s - 744.58s | of these attacks unless you go and demonstrate it. 744.58s - 747.22s | So that is what we did. 747.22s - 749.34s | All right, so to understand how we were able 749.34s - 750.58s | to achieve some of this impact, 750.58s - 752.02s | I want to make sure everyone understands 752.02s - 753.46s | the magical GitHub token. 753.46s - 755.50s | So this is a token that is used 755.50s - 757.38s | by all GitHub Actions workflows 757.38s - 760.74s | to authenticate to GitHub for API or Git operations, 760.74s - 763.82s | and it's an OAuth bearer token that has multiple scopes. 763.82s - 765.42s | Some of these scopes can be read or write, 765.42s - 767.82s | and they're configured at the repository, 767.82s - 769.54s | organization level, or specified 769.54s - 772.10s | within each specific workflow file. 772.10s - 775.26s | So it's very important to know that these tokens 775.26s - 777.74s | are only valid for the duration of each job 777.74s - 778.66s | within a workflow. 778.66s - 781.62s | As soon as that job concludes, the token is no longer valid. 781.62s - 784.78s | So to conduct some of the post-exploitation attacks here, 784.78s - 788.18s | we need to work around this limitation. 788.18s - 790.98s | So let's see how PyTorch, at the time, 790.98s - 794.54s | used or privileged the GitHub token. 794.54s - 797.62s | Okay, so this GitHub token has all the right permissions. 797.62s - 801.58s | This means we have a lot of options to play


chunk_id: 217_chunk_20
chunk_text:
used 755.50s - 757.38s | by all GitHub Actions workflows 757.38s - 760.74s | to authenticate to GitHub for API or Git operations, 760.74s - 763.82s | and it's an OAuth bearer token that has multiple scopes. 763.82s - 765.42s | Some of these scopes can be read or write, 765.42s - 767.82s | and they're configured at the repository, 767.82s - 769.54s | organization level, or specified 769.54s - 772.10s | within each specific workflow file. 772.10s - 775.26s | So it's very important to know that these tokens 775.26s - 777.74s | are only valid for the duration of each job 777.74s - 778.66s | within a workflow. 778.66s - 781.62s | As soon as that job concludes, the token is no longer valid. 781.62s - 784.78s | So to conduct some of the post-exploitation attacks here, 784.78s - 788.18s | we need to work around this limitation. 788.18s - 790.98s | So let's see how PyTorch, at the time, 790.98s - 794.54s | used or privileged the GitHub token. 794.54s - 797.62s | Okay, so this GitHub token has all the right permissions. 797.62s - 801.58s | This means we have a lot of options to play with 801.58s - 803.26s | for some of our post-exploitation, 803.26s - 805.74s | and also, this runner right here is the same one 805.74s - 807.06s | that we now have C2 on. 808.90s - 811.66s | So when a workflow uses the Actions checkout step 811.66s - 814.18s | or some other steps, the GitHub token is stored 814.18s - 816.48s | on the self-hosted runner's file system, 816.48s - 817.82s | which we have compromised. 817.82s - 819.74s | So the problem is that GitHub tokens 819.78s - 822.74s | from fork PRs only have read permissions. 822.74s - 825.34s | So if we tried to take the GitHub token 825.34s - 828.58s | from the fork PR we used to actually implant the runners, 828.58s - 829.90s | it's pretty much useless to us. 829.90s - 832.18s | So the solution is we persist on the runner, 832.18s - 835.38s | and then we capture a token from a future workflow. 835.38s - 836.90s | This is the setup we typically see. 836.90s - 838.78s | We have the workflow from our fork PR, 838.78s - 840.26s | no access to secrets, 840.26s - 843.58s |


chunk_id: 217_chunk_21
chunk_text:
with 801.58s - 803.26s | for some of our post-exploitation, 803.26s - 805.74s | and also, this runner right here is the same one 805.74s - 807.06s | that we now have C2 on. 808.90s - 811.66s | So when a workflow uses the Actions checkout step 811.66s - 814.18s | or some other steps, the GitHub token is stored 814.18s - 816.48s | on the self-hosted runner's file system, 816.48s - 817.82s | which we have compromised. 817.82s - 819.74s | So the problem is that GitHub tokens 819.78s - 822.74s | from fork PRs only have read permissions. 822.74s - 825.34s | So if we tried to take the GitHub token 825.34s - 828.58s | from the fork PR we used to actually implant the runners, 828.58s - 829.90s | it's pretty much useless to us. 829.90s - 832.18s | So the solution is we persist on the runner, 832.18s - 835.38s | and then we capture a token from a future workflow. 835.38s - 836.90s | This is the setup we typically see. 836.90s - 838.78s | We have the workflow from our fork PR, 838.78s - 840.26s | no access to secrets, 840.26s - 843.58s | and a GitHub token with only read permissions. 843.58s - 847.18s | Then we have a workflow from the base repository. 847.18s - 849.90s | This workflow does have access to secrets, 849.90s - 851.62s | and the GitHub token has write permissions. 851.62s - 855.46s | Notice how they both execute on the same self-hosted runner. 855.46s - 857.42s | So this is the reason that persistent, 857.42s - 859.66s | non-ephemeral self-hosted runners are so bad, right? 859.66s - 862.54s | If the runner's ephemeral and we implant it, 862.54s - 865.34s | doesn't give us much access beyond secrets 865.34s - 867.02s | on the file system or something like that. 867.02s - 868.98s | If it's persistent and long-lived, 868.98s - 871.06s | then it will execute future workflows. 871.06s - 873.14s | So we implant the runner, 873.14s - 874.94s | then we wait for future workflows 874.94s - 877.02s | from the base repo to execute on the runner. 877.86s - 879.18s | This is why we need to install persistence 879.18s - 881.38s | rather than just showing code execution. 881.38s - 883.14s | And then we compromise the GitHub token 883.14s -


chunk_id: 217_chunk_22
chunk_text:
and a GitHub token with only read permissions. 843.58s - 847.18s | Then we have a workflow from the base repository. 847.18s - 849.90s | This workflow does have access to secrets, 849.90s - 851.62s | and the GitHub token has write permissions. 851.62s - 855.46s | Notice how they both execute on the same self-hosted runner. 855.46s - 857.42s | So this is the reason that persistent, 857.42s - 859.66s | non-ephemeral self-hosted runners are so bad, right? 859.66s - 862.54s | If the runner's ephemeral and we implant it, 862.54s - 865.34s | doesn't give us much access beyond secrets 865.34s - 867.02s | on the file system or something like that. 867.02s - 868.98s | If it's persistent and long-lived, 868.98s - 871.06s | then it will execute future workflows. 871.06s - 873.14s | So we implant the runner, 873.14s - 874.94s | then we wait for future workflows 874.94s - 877.02s | from the base repo to execute on the runner. 877.86s - 879.18s | This is why we need to install persistence 879.18s - 881.38s | rather than just showing code execution. 881.38s - 883.14s | And then we compromise the GitHub token 883.14s - 886.46s | and any GitHub secrets used by subsequent workflows. 886.46s - 889.14s | So in the PyTorch attack, you can see here, 889.14s - 891.42s | this was our C2 repository. 891.42s - 894.14s | So we were executing code on their self-hosted runners, 894.14s - 895.70s | and we waited for a future workflow 895.70s - 897.82s | to execute from the base repo. 897.82s - 900.70s | And then we just print out the git config file at runtime. 900.70s - 903.90s | So this base64 encoded token is that GitHub token 903.90s - 904.94s | that we were just talking about. 904.94s - 906.74s | So we compromise that GitHub token. 907.42s - 908.26s | And again, we could only use it 908.26s - 910.22s | for the duration of the build, 910.22s - 912.82s | but some of these jobs lasted a long time, 912.82s - 914.62s | so we were able to use it for a while. 914.62s - 916.06s | All right, one of the first things we did 916.06s - 917.22s | after obtaining the GitHub token, 917.22s - 919.06s | which had actions write access, 919.06s - 921.78s | is we


chunk_id: 217_chunk_23
chunk_text:
886.46s | and any GitHub secrets used by subsequent workflows. 886.46s - 889.14s | So in the PyTorch attack, you can see here, 889.14s - 891.42s | this was our C2 repository. 891.42s - 894.14s | So we were executing code on their self-hosted runners, 894.14s - 895.70s | and we waited for a future workflow 895.70s - 897.82s | to execute from the base repo. 897.82s - 900.70s | And then we just print out the git config file at runtime. 900.70s - 903.90s | So this base64 encoded token is that GitHub token 903.90s - 904.94s | that we were just talking about. 904.94s - 906.74s | So we compromise that GitHub token. 907.42s - 908.26s | And again, we could only use it 908.26s - 910.22s | for the duration of the build, 910.22s - 912.82s | but some of these jobs lasted a long time, 912.82s - 914.62s | so we were able to use it for a while. 914.62s - 916.06s | All right, one of the first things we did 916.06s - 917.22s | after obtaining the GitHub token, 917.22s - 919.06s | which had actions write access, 919.06s - 921.78s | is we used it to delete the workflow run logs. 921.78s - 923.34s | Because if we got caught, 923.34s - 925.58s | then we likely wouldn't be able to show impact 925.58s - 926.94s | in order for Meta's triage team 926.94s - 929.18s | to understand the severity of the submission. 930.54s - 931.86s | One of the most direct ways 931.86s - 933.50s | to demonstrate impact with a GitHub token 933.50s - 935.50s | is by modifying GitHub releases. 935.54s - 937.54s | So GitHub releases is just one way 937.54s - 940.22s | for a user to download a release 940.22s - 942.26s | of an application from a repository. 942.26s - 944.42s | What typically will happen is 944.42s - 945.82s | you can upload GitHub releases, 945.82s - 948.70s | so PyTorch uploads releases to their GitHub releases page. 948.70s - 950.22s | And then if you wanna download PyTorch, 950.22s - 951.66s | one of the ways you can do that 951.66s - 954.74s | is by going to GitHub releases and downloading it there. 954.74s - 956.66s | Depending on the permissions of the GitHub token, 956.66s - 960.82s | it can allow


chunk_id: 217_chunk_24
chunk_text:
used it to delete the workflow run logs. 921.78s - 923.34s | Because if we got caught, 923.34s - 925.58s | then we likely wouldn't be able to show impact 925.58s - 926.94s | in order for Meta's triage team 926.94s - 929.18s | to understand the severity of the submission. 930.54s - 931.86s | One of the most direct ways 931.86s - 933.50s | to demonstrate impact with a GitHub token 933.50s - 935.50s | is by modifying GitHub releases. 935.54s - 937.54s | So GitHub releases is just one way 937.54s - 940.22s | for a user to download a release 940.22s - 942.26s | of an application from a repository. 942.26s - 944.42s | What typically will happen is 944.42s - 945.82s | you can upload GitHub releases, 945.82s - 948.70s | so PyTorch uploads releases to their GitHub releases page. 948.70s - 950.22s | And then if you wanna download PyTorch, 950.22s - 951.66s | one of the ways you can do that 951.66s - 954.74s | is by going to GitHub releases and downloading it there. 954.74s - 956.66s | Depending on the permissions of the GitHub token, 956.66s - 960.82s | it can allow you to modify releases and release assets. 960.82s - 963.96s | So you could potentially backdoor a release asset, 963.96s - 965.92s | upload it, and then everyone who downloads 965.92s - 968.76s | and uses that asset will be running your malicious code. 970.00s - 973.24s | We did not want to upload any backdoored assets. 973.24s - 975.32s | We're trying to identify supply chain attacks, 975.32s - 976.52s | not actually execute them. 976.52s - 978.84s | So instead of tampering with assets, 978.84s - 980.52s | we just tampered with the release name. 980.52s - 983.72s | So we sent this curl request using the GitHub API, 983.72s - 985.32s | and it uses the compromised GitHub token 985.32s - 988.72s | just to add my name to the end of the release. 988.72s - 990.52s | So after we sent that request, 990.52s - 993.32s | you can see the latest PyTorch release at the time 993.32s - 995.20s | was now signed by yours truly. 995.20s - 997.52s | And all we did here was just take a screenshot 997.52s - 998.72s | and then revert it right away. 998.72s - 1001.24s | Obviously


chunk_id: 217_chunk_25
chunk_text:
you to modify releases and release assets. 960.82s - 963.96s | So you could potentially backdoor a release asset, 963.96s - 965.92s | upload it, and then everyone who downloads 965.92s - 968.76s | and uses that asset will be running your malicious code. 970.00s - 973.24s | We did not want to upload any backdoored assets. 973.24s - 975.32s | We're trying to identify supply chain attacks, 975.32s - 976.52s | not actually execute them. 976.52s - 978.84s | So instead of tampering with assets, 978.84s - 980.52s | we just tampered with the release name. 980.52s - 983.72s | So we sent this curl request using the GitHub API, 983.72s - 985.32s | and it uses the compromised GitHub token 985.32s - 988.72s | just to add my name to the end of the release. 988.72s - 990.52s | So after we sent that request, 990.52s - 993.32s | you can see the latest PyTorch release at the time 993.32s - 995.20s | was now signed by yours truly. 995.20s - 997.52s | And all we did here was just take a screenshot 997.52s - 998.72s | and then revert it right away. 998.72s - 1001.24s | Obviously we didn't want this being seen by a lot of people, 1001.24s - 1002.84s | but this was our POC to show 1002.84s - 1006.32s | that we could tamper with GitHub releases if we wanted to. 1006.32s - 1007.44s | A problem that we've encountered 1007.44s - 1008.56s | with GitHub release tampering 1008.56s - 1010.70s | is organizations have came back to us 1010.70s - 1011.88s | and legitimately said, 1011.88s - 1015.02s | well, no one really uses our GitHub releases. 1015.02s - 1018.10s | And obviously that shouldn't be the case, 1018.10s - 1020.28s | but to demonstrate further impact, 1020.28s - 1022.56s | we wanted to look at GitHub secrets. 1022.56s - 1025.84s | GitHub secrets are the crown jewels of any repo 1025.84s - 1027.12s | that uses GitHub actions. 1027.12s - 1029.56s | A lot of the times they're overprivileged, 1029.56s - 1032.04s | and they can provide lateral movement opportunities 1032.04s - 1033.76s | beyond the GitHub repository. 1033.76s - 1037.88s | So we were back to being Charlie from It's Always Sunny, 1037.88s - 1039.08s | and we were searching through 1039.08s - 1041.24s | all of those PyTorch workflows again,


chunk_id: 217_chunk_26
chunk_text:
we didn't want this being seen by a lot of people, 1001.24s - 1002.84s | but this was our POC to show 1002.84s - 1006.32s | that we could tamper with GitHub releases if we wanted to. 1006.32s - 1007.44s | A problem that we've encountered 1007.44s - 1008.56s | with GitHub release tampering 1008.56s - 1010.70s | is organizations have came back to us 1010.70s - 1011.88s | and legitimately said, 1011.88s - 1015.02s | well, no one really uses our GitHub releases. 1015.02s - 1018.10s | And obviously that shouldn't be the case, 1018.10s - 1020.28s | but to demonstrate further impact, 1020.28s - 1022.56s | we wanted to look at GitHub secrets. 1022.56s - 1025.84s | GitHub secrets are the crown jewels of any repo 1025.84s - 1027.12s | that uses GitHub actions. 1027.12s - 1029.56s | A lot of the times they're overprivileged, 1029.56s - 1032.04s | and they can provide lateral movement opportunities 1032.04s - 1033.76s | beyond the GitHub repository. 1033.76s - 1037.88s | So we were back to being Charlie from It's Always Sunny, 1037.88s - 1039.08s | and we were searching through 1039.08s - 1041.24s | all of those PyTorch workflows again, 1041.24s - 1043.00s | trying to see where GitHub secrets were used 1043.00s - 1044.48s | and see if we could compromise them. 1044.48s - 1046.24s | Here, you can see that they're using 1046.24s - 1047.60s | some personal access tokens, 1047.60s - 1048.80s | which are always super interesting. 1048.80s - 1050.38s | They're using AWS keys, 1050.38s - 1051.96s | and there were a bunch of other secrets 1052.48s - 1053.76s | used by the PyTorch repository. 1053.76s - 1055.64s | And before we go any further, 1055.64s - 1057.60s | I want to quickly recap how we got here. 1057.60s - 1060.78s | So remembering back to the grammar police, 1060.78s - 1062.50s | we came, we fixed the typo, 1062.50s - 1064.02s | that typo got merged, 1064.02s - 1066.40s | that made us a contributor to the repository, 1066.40s - 1067.80s | which was using the default setting 1067.80s - 1069.96s | so we could execute arbitrary workflows. 1069.96s - 1072.20s | We used that access to execute a workflow 1072.20s - 1074.84s | that implanted three of their self-hosted runners. 1074.84s - 1076.60s | Then we stole this GitHub token 1076.60s - 1080.24s


chunk_id: 217_chunk_27
chunk_text:
1041.24s - 1043.00s | trying to see where GitHub secrets were used 1043.00s - 1044.48s | and see if we could compromise them. 1044.48s - 1046.24s | Here, you can see that they're using 1046.24s - 1047.60s | some personal access tokens, 1047.60s - 1048.80s | which are always super interesting. 1048.80s - 1050.38s | They're using AWS keys, 1050.38s - 1051.96s | and there were a bunch of other secrets 1052.48s - 1053.76s | used by the PyTorch repository. 1053.76s - 1055.64s | And before we go any further, 1055.64s - 1057.60s | I want to quickly recap how we got here. 1057.60s - 1060.78s | So remembering back to the grammar police, 1060.78s - 1062.50s | we came, we fixed the typo, 1062.50s - 1064.02s | that typo got merged, 1064.02s - 1066.40s | that made us a contributor to the repository, 1066.40s - 1067.80s | which was using the default setting 1067.80s - 1069.96s | so we could execute arbitrary workflows. 1069.96s - 1072.20s | We used that access to execute a workflow 1072.20s - 1074.84s | that implanted three of their self-hosted runners. 1074.84s - 1076.60s | Then we stole this GitHub token 1076.60s - 1080.24s | from a future workflow executing on that runner, 1080.24s - 1082.24s | and now we're going to try to use this GitHub token 1082.24s - 1084.60s | to exfiltrate GitHub secrets. 1084.60s - 1086.96s | All right, one problem that we had to work around 1086.96s - 1089.58s | is that the workflows that had the GitHub secrets 1089.58s - 1090.92s | that we wanted to take 1090.92s - 1092.92s | didn't actually run on the self-hosted runners 1092.92s - 1094.12s | that we were now sitting on. 1094.12s - 1097.04s | So the solution here was to use the GitHub token 1097.04s - 1099.40s | to conduct pipeline privilege escalation 1099.40s - 1102.76s | and get arbitrary code execution within those workflows 1102.76s - 1105.24s | so we could get those very nice secrets. 1106.36s - 1108.60s | Another problem is GitHub tokens 1108.60s - 1110.20s | aren't allowed to modify files 1110.20s - 1112.80s | in the .github-workflows directory. 1112.80s - 1115.20s | That seems like a kind of random restriction by GitHub, 1115.20s - 1117.48s | but it's actually implemented a few years ago 1117.48s - 1119.68s | to prevent these exact attacks. 1119.68s - 1121.10s


chunk_id: 217_chunk_28
chunk_text:
| from a future workflow executing on that runner, 1080.24s - 1082.24s | and now we're going to try to use this GitHub token 1082.24s - 1084.60s | to exfiltrate GitHub secrets. 1084.60s - 1086.96s | All right, one problem that we had to work around 1086.96s - 1089.58s | is that the workflows that had the GitHub secrets 1089.58s - 1090.92s | that we wanted to take 1090.92s - 1092.92s | didn't actually run on the self-hosted runners 1092.92s - 1094.12s | that we were now sitting on. 1094.12s - 1097.04s | So the solution here was to use the GitHub token 1097.04s - 1099.40s | to conduct pipeline privilege escalation 1099.40s - 1102.76s | and get arbitrary code execution within those workflows 1102.76s - 1105.24s | so we could get those very nice secrets. 1106.36s - 1108.60s | Another problem is GitHub tokens 1108.60s - 1110.20s | aren't allowed to modify files 1110.20s - 1112.80s | in the .github-workflows directory. 1112.80s - 1115.20s | That seems like a kind of random restriction by GitHub, 1115.20s - 1117.48s | but it's actually implemented a few years ago 1117.48s - 1119.68s | to prevent these exact attacks. 1119.68s - 1121.10s | So they want to stop attackers 1121.10s - 1123.08s | from doing what we're trying to do right now, 1123.08s - 1124.92s | and so you can't just make a new branch, 1124.92s - 1126.72s | modify a workflow to exfiltrate secrets, 1126.72s - 1128.70s | and then you're done. 1128.70s - 1130.96s | The solution here is you need to find a workflow 1130.96s - 1133.38s | that uses the GitHub secrets you're interested in 1133.38s - 1135.34s | that executes code from outside 1135.34s - 1137.48s | of this .github-workflows directory. 1137.48s - 1139.80s | So this was our path here. 1139.80s - 1142.48s | If you look back at PyTorch's weekly.yaml workflow 1142.48s - 1146.72s | at the time, you can see it does two interesting things. 1146.72s - 1148.48s | One of them is that it calls this 1148.48s - 1151.32s | underscore update commit hash .yaml workflow. 1151.32s - 1154.76s | The other is that it uses these update bot tokens 1154.76s - 1157.08s | and PyTorch bot tokens, which based on context, 1157.08s - 1159.80s | we were pretty sure were GitHub personal access tokens. 1159.80s - 1162.56s | So let's look at


chunk_id: 217_chunk_29
chunk_text:
| So they want to stop attackers 1121.10s - 1123.08s | from doing what we're trying to do right now, 1123.08s - 1124.92s | and so you can't just make a new branch, 1124.92s - 1126.72s | modify a workflow to exfiltrate secrets, 1126.72s - 1128.70s | and then you're done. 1128.70s - 1130.96s | The solution here is you need to find a workflow 1130.96s - 1133.38s | that uses the GitHub secrets you're interested in 1133.38s - 1135.34s | that executes code from outside 1135.34s - 1137.48s | of this .github-workflows directory. 1137.48s - 1139.80s | So this was our path here. 1139.80s - 1142.48s | If you look back at PyTorch's weekly.yaml workflow 1142.48s - 1146.72s | at the time, you can see it does two interesting things. 1146.72s - 1148.48s | One of them is that it calls this 1148.48s - 1151.32s | underscore update commit hash .yaml workflow. 1151.32s - 1154.76s | The other is that it uses these update bot tokens 1154.76s - 1157.08s | and PyTorch bot tokens, which based on context, 1157.08s - 1159.80s | we were pretty sure were GitHub personal access tokens. 1159.80s - 1162.56s | So let's look at update commit hash .yaml. 1163.76s - 1166.88s | This workflow is still in the restricted workflows directory 1166.88s - 1170.44s | so we can't modify this workflow with the GitHub token, 1170.44s - 1174.24s | but it calls update commit hashes .py, 1174.24s - 1176.64s | which is in the .github-scripts directory, 1176.64s - 1179.40s | so it's not in the restricted workflow directory. 1179.40s - 1180.92s | All right, so all we need to do now 1180.92s - 1184.08s | is use the GitHub token to make a new feature branch 1184.08s - 1187.84s | and modify that script, and we know that the workflow 1187.84s - 1189.68s | that calls it runs on a certain event 1189.68s - 1191.24s | so we can take advantage of it. 1192.72s - 1194.76s | The modification that we introduced was quite simple. 1194.76s - 1197.04s | We saved off the tokens that it received 1197.04s - 1198.80s | via environment variables to files, 1198.80s - 1201.48s | and then we used our public key to encrypt them 1201.48s - 1203.36s | and print them to the build logs. 1203.36s - 1205.24s | This would prevent any other attackers 1205.24s - 1208.52s


chunk_id: 217_chunk_30
chunk_text:
update commit hash .yaml. 1163.76s - 1166.88s | This workflow is still in the restricted workflows directory 1166.88s - 1170.44s | so we can't modify this workflow with the GitHub token, 1170.44s - 1174.24s | but it calls update commit hashes .py, 1174.24s - 1176.64s | which is in the .github-scripts directory, 1176.64s - 1179.40s | so it's not in the restricted workflow directory. 1179.40s - 1180.92s | All right, so all we need to do now 1180.92s - 1184.08s | is use the GitHub token to make a new feature branch 1184.08s - 1187.84s | and modify that script, and we know that the workflow 1187.84s - 1189.68s | that calls it runs on a certain event 1189.68s - 1191.24s | so we can take advantage of it. 1192.72s - 1194.76s | The modification that we introduced was quite simple. 1194.76s - 1197.04s | We saved off the tokens that it received 1197.04s - 1198.80s | via environment variables to files, 1198.80s - 1201.48s | and then we used our public key to encrypt them 1201.48s - 1203.36s | and print them to the build logs. 1203.36s - 1205.24s | This would prevent any other attackers 1205.24s - 1208.52s | from getting those secrets while we were conducting our POC. 1209.72s - 1210.80s | Okay, so what did we actually do? 1210.80s - 1213.76s | We used the GitHub token to create a new branch. 1213.76s - 1215.48s | Then on the new branch, we added our payload 1215.48s - 1218.12s | to update commit hashes .py script. 1218.12s - 1220.24s | Then we used the GitHub token to trigger our payload 1220.24s - 1222.12s | via a workflow dispatch event. 1222.12s - 1224.96s | This is because the token had actions write privileges, 1224.96s - 1226.96s | and then we could retrieve the encrypted secrets 1226.96s - 1229.16s | from the build logs, cancel the workflow, 1229.16s - 1231.76s | delete the workflow logs, and actually decrypt the secrets. 1231.76s - 1234.96s | So at the time when we were doing this operation, 1234.96s - 1237.44s | I was absolutely terrified that I was gonna forget 1237.44s - 1240.24s | to take a screenshot, so instead of taking screenshots, 1240.24s - 1241.68s | I just screen-recorded the whole thing, 1241.68s - 1243.52s | so I have like an eight-hour screen recording 1243.52s - 1245.40s | of all this


chunk_id: 217_chunk_31
chunk_text:
| from getting those secrets while we were conducting our POC. 1209.72s - 1210.80s | Okay, so what did we actually do? 1210.80s - 1213.76s | We used the GitHub token to create a new branch. 1213.76s - 1215.48s | Then on the new branch, we added our payload 1215.48s - 1218.12s | to update commit hashes .py script. 1218.12s - 1220.24s | Then we used the GitHub token to trigger our payload 1220.24s - 1222.12s | via a workflow dispatch event. 1222.12s - 1224.96s | This is because the token had actions write privileges, 1224.96s - 1226.96s | and then we could retrieve the encrypted secrets 1226.96s - 1229.16s | from the build logs, cancel the workflow, 1229.16s - 1231.76s | delete the workflow logs, and actually decrypt the secrets. 1231.76s - 1234.96s | So at the time when we were doing this operation, 1234.96s - 1237.44s | I was absolutely terrified that I was gonna forget 1237.44s - 1240.24s | to take a screenshot, so instead of taking screenshots, 1240.24s - 1241.68s | I just screen-recorded the whole thing, 1241.68s - 1243.52s | so I have like an eight-hour screen recording 1243.52s - 1245.40s | of all this stuff from PyTorch, 1245.40s - 1248.20s | and I condensed some of it into a video on the next slide 1248.20s - 1251.32s | just showing the GitHub personal access token exfiltration, 1251.32s - 1253.12s | so we'll walk you all through that. 1254.24s - 1256.56s | Here we have our run-on-runner C2, 1256.56s - 1259.44s | and we're gonna navigate to our command output, 1259.44s - 1263.32s | and we're just gonna get the base64-encoded GitHub token. 1263.32s - 1265.60s | This is from a future workflow that was executing 1265.60s - 1267.08s | on one of the compromised runners. 1267.08s - 1270.04s | So we're gonna base64 decode that token, 1270.04s - 1271.92s | and you can see it's just an authorization header 1271.92s - 1273.52s | within the GitHub token. 1273.52s - 1275.60s | Use the token to clone the repository, 1275.60s - 1278.64s | and then we made a new branch in the PyTorch repository. 1278.64s - 1281.96s | So within our new branch, we're going to modify 1281.96s - 1283.72s | that Python script that we identified 1283.72s - 1285.20s | as an injection point. 1285.20s - 1288.12s | Here with my really poor Vim


chunk_id: 217_chunk_32
chunk_text:
stuff from PyTorch, 1245.40s - 1248.20s | and I condensed some of it into a video on the next slide 1248.20s - 1251.32s | just showing the GitHub personal access token exfiltration, 1251.32s - 1253.12s | so we'll walk you all through that. 1254.24s - 1256.56s | Here we have our run-on-runner C2, 1256.56s - 1259.44s | and we're gonna navigate to our command output, 1259.44s - 1263.32s | and we're just gonna get the base64-encoded GitHub token. 1263.32s - 1265.60s | This is from a future workflow that was executing 1265.60s - 1267.08s | on one of the compromised runners. 1267.08s - 1270.04s | So we're gonna base64 decode that token, 1270.04s - 1271.92s | and you can see it's just an authorization header 1271.92s - 1273.52s | within the GitHub token. 1273.52s - 1275.60s | Use the token to clone the repository, 1275.60s - 1278.64s | and then we made a new branch in the PyTorch repository. 1278.64s - 1281.96s | So within our new branch, we're going to modify 1281.96s - 1283.72s | that Python script that we identified 1283.72s - 1285.20s | as an injection point. 1285.20s - 1288.12s | Here with my really poor Vim skills, 1288.12s - 1289.48s | I'm gonna paste our new payload 1289.48s - 1291.20s | to exfiltrate those secrets, 1291.20s - 1293.76s | and then once the payload is set, 1293.76s - 1295.68s | we're gonna actually commit this to the repository 1295.68s - 1297.68s | because the token has contents right. 1297.68s - 1299.36s | So we committed to the repo. 1299.36s - 1301.32s | Now we're gonna use a curl request 1301.32s - 1304.04s | just to trigger this workflow from our branch 1304.04s - 1305.88s | which has the injected code in it, 1305.88s - 1307.16s | and then we're looking at 1307.16s - 1308.72s | the actual PyTorch repository now 1308.72s - 1310.48s | where we just triggered a workflow, 1310.48s - 1315.16s | and we're gonna see two base64-encoded encrypted blobs. 1315.16s - 1316.64s | So we're gonna scroll down, 1316.64s - 1317.96s | and we're gonna get these blobs 1317.96s - 1319.72s | which is the output of our commands, 1319.72s - 1323.60s | and then we're gonna go to this other terminal 1323.60s - 1324.92s | and decrypt them. 1324.92s - 1327.04s | So first, we're just gonna base64 decode them, 1327.04s -


chunk_id: 217_chunk_33
chunk_text:
skills, 1288.12s - 1289.48s | I'm gonna paste our new payload 1289.48s - 1291.20s | to exfiltrate those secrets, 1291.20s - 1293.76s | and then once the payload is set, 1293.76s - 1295.68s | we're gonna actually commit this to the repository 1295.68s - 1297.68s | because the token has contents right. 1297.68s - 1299.36s | So we committed to the repo. 1299.36s - 1301.32s | Now we're gonna use a curl request 1301.32s - 1304.04s | just to trigger this workflow from our branch 1304.04s - 1305.88s | which has the injected code in it, 1305.88s - 1307.16s | and then we're looking at 1307.16s - 1308.72s | the actual PyTorch repository now 1308.72s - 1310.48s | where we just triggered a workflow, 1310.48s - 1315.16s | and we're gonna see two base64-encoded encrypted blobs. 1315.16s - 1316.64s | So we're gonna scroll down, 1316.64s - 1317.96s | and we're gonna get these blobs 1317.96s - 1319.72s | which is the output of our commands, 1319.72s - 1323.60s | and then we're gonna go to this other terminal 1323.60s - 1324.92s | and decrypt them. 1324.92s - 1327.04s | So first, we're just gonna base64 decode them, 1327.04s - 1328.68s | and then we're gonna save them to a file, 1328.68s - 1330.76s | and we're gonna use our private key locally 1330.76s - 1332.52s | to actually decrypt them and get the token. 1332.52s - 1335.80s | So once you see some output that starts 1335.84s - 1337.64s | with G-H-P underscore, 1337.64s - 1339.24s | those are the personal access tokens, 1339.24s - 1342.96s | and we're gonna take them and run them through Gato 1342.96s - 1345.20s | which will do a bunch of automated enumeration 1345.20s - 1346.88s | using the GitHub APIs, 1346.88s - 1349.72s | and then Gato will tell us who the owners are, 1349.72s - 1350.80s | what they have access to, 1350.80s - 1352.88s | and what the potential impact is 1352.88s - 1356.00s | because if they just had read access to the repository, 1356.00s - 1357.96s | there's not a real risk here to PyTorch. 1357.96s - 1361.16s | If they have other more sensitive privileges, 1361.16s - 1362.40s | then it could be bad. 1362.40s - 1364.20s | So here we're running them through Gato. 1364.24s - 1365.40s | You can see one of the users 1365.40s


chunk_id: 217_chunk_34
chunk_text:
1328.68s | and then we're gonna save them to a file, 1328.68s - 1330.76s | and we're gonna use our private key locally 1330.76s - 1332.52s | to actually decrypt them and get the token. 1332.52s - 1335.80s | So once you see some output that starts 1335.84s - 1337.64s | with G-H-P underscore, 1337.64s - 1339.24s | those are the personal access tokens, 1339.24s - 1342.96s | and we're gonna take them and run them through Gato 1342.96s - 1345.20s | which will do a bunch of automated enumeration 1345.20s - 1346.88s | using the GitHub APIs, 1346.88s - 1349.72s | and then Gato will tell us who the owners are, 1349.72s - 1350.80s | what they have access to, 1350.80s - 1352.88s | and what the potential impact is 1352.88s - 1356.00s | because if they just had read access to the repository, 1356.00s - 1357.96s | there's not a real risk here to PyTorch. 1357.96s - 1361.16s | If they have other more sensitive privileges, 1361.16s - 1362.40s | then it could be bad. 1362.40s - 1364.20s | So here we're running them through Gato. 1364.24s - 1365.40s | You can see one of the users 1365.40s - 1367.00s | is the PyTorch update bot user, 1367.00s - 1369.08s | the other is the PyTorch bot user, 1369.08s - 1371.24s | and they're both PyTorch organization members. 1371.24s - 1373.20s | The tokens have repo scope, 1373.20s - 1375.12s | so we can use this now to interact 1375.12s - 1376.88s | with the PyTorch repository directly. 1376.88s - 1378.52s | These are long-lived credentials, 1378.52s - 1380.68s | and they had access to 93 repos 1380.68s - 1382.40s | within the PyTorch organization, 1382.40s - 1383.60s | a lot of which were private 1383.60s - 1385.08s | and did not have branch protection, 1385.08s - 1387.36s | so you could have done whatever you wanted to them, 1387.36s - 1391.48s | and there's a lot of paths here to supply chain compromise. 1391.48s - 1392.96s | All right, that was a lot, right? 1392.96s - 1395.04s | Okay, so what can we do? 1395.04s - 1396.80s | So we have those two tokens, 1396.80s - 1398.52s | so we can use them by themselves 1398.52s - 1400.64s | to simply introduce code into the main branch. 1400.64s - 1401.96s | One creates a pull request 1401.96s


chunk_id: 217_chunk_35
chunk_text:
- 1367.00s | is the PyTorch update bot user, 1367.00s - 1369.08s | the other is the PyTorch bot user, 1369.08s - 1371.24s | and they're both PyTorch organization members. 1371.24s - 1373.20s | The tokens have repo scope, 1373.20s - 1375.12s | so we can use this now to interact 1375.12s - 1376.88s | with the PyTorch repository directly. 1376.88s - 1378.52s | These are long-lived credentials, 1378.52s - 1380.68s | and they had access to 93 repos 1380.68s - 1382.40s | within the PyTorch organization, 1382.40s - 1383.60s | a lot of which were private 1383.60s - 1385.08s | and did not have branch protection, 1385.08s - 1387.36s | so you could have done whatever you wanted to them, 1387.36s - 1391.48s | and there's a lot of paths here to supply chain compromise. 1391.48s - 1392.96s | All right, that was a lot, right? 1392.96s - 1395.04s | Okay, so what can we do? 1395.04s - 1396.80s | So we have those two tokens, 1396.80s - 1398.52s | so we can use them by themselves 1398.52s - 1400.64s | to simply introduce code into the main branch. 1400.64s - 1401.96s | One creates a pull request 1401.96s - 1403.68s | and have the other one approve and merge it. 1403.68s - 1405.60s | Boom, supply chain compromise. 1405.60s - 1408.20s | Another form would be to backdoor a dependency 1408.20s - 1409.60s | within the PyTorch organization 1409.60s - 1412.16s | that's used by the primary PyTorch repo, 1412.16s - 1413.84s | and that could be a more subtle way 1413.84s - 1417.04s | to obtain that same final impact. 1417.04s - 1419.04s | And finally, there's even more clever ways, 1419.04s - 1422.56s | like smuggling malicious code into an in-progress feature 1423.00s - 1424.20s | that a developer is working on 1424.20s - 1425.36s | and then have that merged in. 1425.36s - 1427.96s | So really, there's too many paths to count 1427.96s - 1430.88s | for how this could impact end users of PyTorch. 1432.60s - 1434.92s | If you want to see even more impact 1434.92s - 1436.88s | and you're not really into GitHub, 1436.88s - 1438.24s | this section is for you. 1438.24s - 1440.36s | So if you noticed a few slides ago 1440.36s - 1441.52s | when we were looking at their secrets, 1441.52s - 1444.04s | we


chunk_id: 217_chunk_36
chunk_text:
- 1403.68s | and have the other one approve and merge it. 1403.68s - 1405.60s | Boom, supply chain compromise. 1405.60s - 1408.20s | Another form would be to backdoor a dependency 1408.20s - 1409.60s | within the PyTorch organization 1409.60s - 1412.16s | that's used by the primary PyTorch repo, 1412.16s - 1413.84s | and that could be a more subtle way 1413.84s - 1417.04s | to obtain that same final impact. 1417.04s - 1419.04s | And finally, there's even more clever ways, 1419.04s - 1422.56s | like smuggling malicious code into an in-progress feature 1423.00s - 1424.20s | that a developer is working on 1424.20s - 1425.36s | and then have that merged in. 1425.36s - 1427.96s | So really, there's too many paths to count 1427.96s - 1430.88s | for how this could impact end users of PyTorch. 1432.60s - 1434.92s | If you want to see even more impact 1434.92s - 1436.88s | and you're not really into GitHub, 1436.88s - 1438.24s | this section is for you. 1438.24s - 1440.36s | So if you noticed a few slides ago 1440.36s - 1441.52s | when we were looking at their secrets, 1441.52s - 1444.04s | we saw that they also had AWS keys. 1444.04s - 1446.72s | We conducted a very similar attack 1446.72s - 1448.12s | to the one we just showed, right, 1448.12s - 1449.84s | where we do that kind of workflow hopping 1449.84s - 1452.04s | and injection and triggering with the GitHub token. 1452.04s - 1454.80s | We repeated that process to grab their AWS keys. 1454.80s - 1456.24s | And the reason we wanted to do this 1456.24s - 1458.48s | was if you don't understand GitHub, 1458.48s - 1460.84s | it can sometimes be hard to even see 1460.84s - 1464.76s | why code contributions to the main branch of a repository 1464.76s - 1465.76s | could be impactful. 1465.76s - 1469.04s | But everyone understands AWS and AWS releases. 1469.04s - 1470.52s | So we grabbed these keys, 1470.52s - 1472.86s | then we used the AWS CLI to authenticate, 1472.86s - 1475.64s | and we confirmed that we were the PyTorch bot user. 1475.64s - 1477.48s | So we didn't want to poke around too much, 1477.48s - 1480.68s | but we did some basic AWS S3LS commands 1480.68s - 1483.36s | and saw that this user


chunk_id: 217_chunk_37
chunk_text:
saw that they also had AWS keys. 1444.04s - 1446.72s | We conducted a very similar attack 1446.72s - 1448.12s | to the one we just showed, right, 1448.12s - 1449.84s | where we do that kind of workflow hopping 1449.84s - 1452.04s | and injection and triggering with the GitHub token. 1452.04s - 1454.80s | We repeated that process to grab their AWS keys. 1454.80s - 1456.24s | And the reason we wanted to do this 1456.24s - 1458.48s | was if you don't understand GitHub, 1458.48s - 1460.84s | it can sometimes be hard to even see 1460.84s - 1464.76s | why code contributions to the main branch of a repository 1464.76s - 1465.76s | could be impactful. 1465.76s - 1469.04s | But everyone understands AWS and AWS releases. 1469.04s - 1470.52s | So we grabbed these keys, 1470.52s - 1472.86s | then we used the AWS CLI to authenticate, 1472.86s - 1475.64s | and we confirmed that we were the PyTorch bot user. 1475.64s - 1477.48s | So we didn't want to poke around too much, 1477.48s - 1480.68s | but we did some basic AWS S3LS commands 1480.68s - 1483.36s | and saw that this user had a lot of privileges. 1483.36s - 1485.16s | We confirmed through the workflow logs 1485.16s - 1488.08s | that they could actually write to these S3 buckets. 1488.08s - 1491.08s | And one weird thing that we weren't really expecting to see 1491.08s - 1494.56s | was there were PyTorch releases in these S3 buckets. 1494.56s - 1498.16s | So at the time, honestly, I didn't think too much of it 1498.16s - 1500.12s | because I was kind of freaked out with the access we had 1500.12s - 1502.28s | and wanted to submit the report right away. 1502.28s - 1503.12s | I think, like we said, 1503.12s - 1505.72s | this was hour eight or nine at this point, 1505.72s - 1507.20s | and I just wanted to get the report in. 1507.20s - 1509.36s | So I didn't really pay attention to these releases. 1509.36s - 1511.56s | And then we were rehearsing like a week ago 1511.56s - 1513.40s | and we were talking and I was like, 1513.40s - 1516.12s | why were these releases in these S3 buckets? 1516.12s - 1517.96s | Like what system was pulling releases from here? 1517.96s - 1521.24s


chunk_id: 217_chunk_38
chunk_text:
had a lot of privileges. 1483.36s - 1485.16s | We confirmed through the workflow logs 1485.16s - 1488.08s | that they could actually write to these S3 buckets. 1488.08s - 1491.08s | And one weird thing that we weren't really expecting to see 1491.08s - 1494.56s | was there were PyTorch releases in these S3 buckets. 1494.56s - 1498.16s | So at the time, honestly, I didn't think too much of it 1498.16s - 1500.12s | because I was kind of freaked out with the access we had 1500.12s - 1502.28s | and wanted to submit the report right away. 1502.28s - 1503.12s | I think, like we said, 1503.12s - 1505.72s | this was hour eight or nine at this point, 1505.72s - 1507.20s | and I just wanted to get the report in. 1507.20s - 1509.36s | So I didn't really pay attention to these releases. 1509.36s - 1511.56s | And then we were rehearsing like a week ago 1511.56s - 1513.40s | and we were talking and I was like, 1513.40s - 1516.12s | why were these releases in these S3 buckets? 1516.12s - 1517.96s | Like what system was pulling releases from here? 1517.96s - 1521.24s | So I went to the PyTorch website 1521.24s - 1525.32s | and just going to PyTorch.org or PyTorch.com, 1525.32s - 1526.68s | whatever it is, you can scroll down 1526.68s - 1528.60s | and you can see when they instruct users 1528.60s - 1531.00s | to install, download and install PyTorch, 1531.00s - 1532.76s | if you're using PIP or whatever, 1532.76s - 1536.96s | you're using this download dot, what is it? 1536.96s - 1538.92s | Download.pytorch.org URL. 1539.44s - 1541.36s | So a lot of the releases were coming from this URL. 1541.36s - 1544.84s | So we went to this URL in a new tab 1544.84s - 1548.20s | and we're just looking at the request and response headers 1548.20s - 1551.16s | and sure enough, in the headers, 1551.16s - 1553.60s | they confirmed that these were actually pulling releases 1553.60s - 1555.80s | from these S3 buckets 1555.80s - 1560.68s | and the layout of the release assets on that website 1560.68s - 1563.04s | were identical to the layout in the S3 buckets. 1563.04s - 1565.52s | So basically with those AWS keys, 1565.52s - 1567.92s | we could have uploaded our own


chunk_id: 217_chunk_39
chunk_text:
| So I went to the PyTorch website 1521.24s - 1525.32s | and just going to PyTorch.org or PyTorch.com, 1525.32s - 1526.68s | whatever it is, you can scroll down 1526.68s - 1528.60s | and you can see when they instruct users 1528.60s - 1531.00s | to install, download and install PyTorch, 1531.00s - 1532.76s | if you're using PIP or whatever, 1532.76s - 1536.96s | you're using this download dot, what is it? 1536.96s - 1538.92s | Download.pytorch.org URL. 1539.44s - 1541.36s | So a lot of the releases were coming from this URL. 1541.36s - 1544.84s | So we went to this URL in a new tab 1544.84s - 1548.20s | and we're just looking at the request and response headers 1548.20s - 1551.16s | and sure enough, in the headers, 1551.16s - 1553.60s | they confirmed that these were actually pulling releases 1553.60s - 1555.80s | from these S3 buckets 1555.80s - 1560.68s | and the layout of the release assets on that website 1560.68s - 1563.04s | were identical to the layout in the S3 buckets. 1563.04s - 1565.52s | So basically with those AWS keys, 1565.52s - 1567.92s | we could have uploaded our own PyTorch releases 1567.92s - 1572.92s | to these S3 buckets and then anyone who downloads PyTorch 1573.16s - 1575.24s | by following the instructions from the website 1575.24s - 1576.40s | is going to be downloading 1576.40s - 1578.48s | and potentially executing our malicious code. 1578.48s - 1581.56s | So this is that kind of SolarWinds style supply chain attack 1581.56s - 1585.64s | for anyone that uses PyTorch in the future. 1585.64s - 1587.64s | Okay, that was a lot. 1587.64s - 1588.96s | We just did a lot. 1588.96s - 1590.10s | This is what it looks like 1590.10s - 1592.30s | put into a nice little red team diagram 1592.30s - 1595.72s | and if anyone got here late or fell asleep, 1595.72s - 1596.68s | we'll do a quick recap. 1596.72s - 1599.20s | So we started by opening a typo fix PR 1599.20s - 1601.20s | in the PyTorch repository. 1601.20s - 1604.20s | That PR got merged and then we were a contributor 1604.20s - 1606.38s | so we submitted a malicious pull request 1606.38s - 1609.04s | that installed our runner on runner C2 1609.04s - 1612.08s | on three different self-hosted


chunk_id: 217_chunk_40
chunk_text:
PyTorch releases 1567.92s - 1572.92s | to these S3 buckets and then anyone who downloads PyTorch 1573.16s - 1575.24s | by following the instructions from the website 1575.24s - 1576.40s | is going to be downloading 1576.40s - 1578.48s | and potentially executing our malicious code. 1578.48s - 1581.56s | So this is that kind of SolarWinds style supply chain attack 1581.56s - 1585.64s | for anyone that uses PyTorch in the future. 1585.64s - 1587.64s | Okay, that was a lot. 1587.64s - 1588.96s | We just did a lot. 1588.96s - 1590.10s | This is what it looks like 1590.10s - 1592.30s | put into a nice little red team diagram 1592.30s - 1595.72s | and if anyone got here late or fell asleep, 1595.72s - 1596.68s | we'll do a quick recap. 1596.72s - 1599.20s | So we started by opening a typo fix PR 1599.20s - 1601.20s | in the PyTorch repository. 1601.20s - 1604.20s | That PR got merged and then we were a contributor 1604.20s - 1606.38s | so we submitted a malicious pull request 1606.38s - 1609.04s | that installed our runner on runner C2 1609.04s - 1612.08s | on three different self-hosted GitHub runners. 1612.08s - 1613.12s | From one of these runners, 1613.12s - 1615.88s | we stole a GitHub token from a future build 1615.88s - 1617.28s | and we used that to do a bunch of stuff. 1617.28s - 1619.16s | First, we just modified that release 1619.16s - 1622.40s | by updating the release title with my name and reverting. 1622.40s - 1624.40s | Then we created a feature branch 1624.40s - 1626.08s | in the PyTorch repository 1626.08s - 1628.20s | and we triggered additional workflows 1628.20s - 1630.26s | using that workflow dispatch event 1630.26s - 1631.64s | injecting through the scripts 1631.64s - 1635.12s | that were outside of the restricted workflows directory. 1635.12s - 1637.24s | Through this process, we stole some secrets. 1637.24s - 1639.64s | We stole the PyTorch bot AWS keys 1639.64s - 1642.08s | and then we stole two GitHub personal access tokens 1642.08s - 1644.00s | used by PyTorch bot users. 1644.00s - 1645.24s | There were a bunch more secrets in there 1645.24s - 1648.08s | but we didn't think there was any need 1648.08s - 1650.52s | for additional demonstration of impact. 1650.52s - 1651.64s | Using


chunk_id: 217_chunk_41
chunk_text:
GitHub runners. 1612.08s - 1613.12s | From one of these runners, 1613.12s - 1615.88s | we stole a GitHub token from a future build 1615.88s - 1617.28s | and we used that to do a bunch of stuff. 1617.28s - 1619.16s | First, we just modified that release 1619.16s - 1622.40s | by updating the release title with my name and reverting. 1622.40s - 1624.40s | Then we created a feature branch 1624.40s - 1626.08s | in the PyTorch repository 1626.08s - 1628.20s | and we triggered additional workflows 1628.20s - 1630.26s | using that workflow dispatch event 1630.26s - 1631.64s | injecting through the scripts 1631.64s - 1635.12s | that were outside of the restricted workflows directory. 1635.12s - 1637.24s | Through this process, we stole some secrets. 1637.24s - 1639.64s | We stole the PyTorch bot AWS keys 1639.64s - 1642.08s | and then we stole two GitHub personal access tokens 1642.08s - 1644.00s | used by PyTorch bot users. 1644.00s - 1645.24s | There were a bunch more secrets in there 1645.24s - 1648.08s | but we didn't think there was any need 1648.08s - 1650.52s | for additional demonstration of impact. 1650.52s - 1651.64s | Using the AWS keys, 1651.64s - 1655.02s | we could upload new PyTorch releases to this S3 bucket. 1655.02s - 1656.66s | Obviously, we did not do that. 1656.66s - 1659.14s | Then using the GitHub personal access tokens, 1659.14s - 1661.34s | we could have modified PyTorch dependencies. 1661.34s - 1662.68s | We could have used them in conjunction 1662.68s - 1665.70s | to commit and then approve merges to main 1665.70s - 1668.00s | and done a lot of other bad stuff. 1668.00s - 1669.66s | All of this goes to say, at the end of the day, 1669.66s - 1672.48s | there were a lot of supply chain compromised paths here 1672.48s - 1677.22s | which would have affected most people using PyTorch 1677.22s - 1679.02s | both currently and in the future. 1680.42s - 1683.52s | You would think that this amount of demonstration of impact 1683.52s - 1685.40s | would be enough to convince PyTorch 1685.40s - 1689.08s | to immediately and urgently apply fixes 1689.08s - 1692.64s | and well, we'll let the disclosure timeline 1692.64s - 1693.52s | speak for itself. 1693.52s - 1697.40s | So we submitted this issue in August 2023 1697.40s


chunk_id: 217_chunk_42
chunk_text:
the AWS keys, 1651.64s - 1655.02s | we could upload new PyTorch releases to this S3 bucket. 1655.02s - 1656.66s | Obviously, we did not do that. 1656.66s - 1659.14s | Then using the GitHub personal access tokens, 1659.14s - 1661.34s | we could have modified PyTorch dependencies. 1661.34s - 1662.68s | We could have used them in conjunction 1662.68s - 1665.70s | to commit and then approve merges to main 1665.70s - 1668.00s | and done a lot of other bad stuff. 1668.00s - 1669.66s | All of this goes to say, at the end of the day, 1669.66s - 1672.48s | there were a lot of supply chain compromised paths here 1672.48s - 1677.22s | which would have affected most people using PyTorch 1677.22s - 1679.02s | both currently and in the future. 1680.42s - 1683.52s | You would think that this amount of demonstration of impact 1683.52s - 1685.40s | would be enough to convince PyTorch 1685.40s - 1689.08s | to immediately and urgently apply fixes 1689.08s - 1692.64s | and well, we'll let the disclosure timeline 1692.64s - 1693.52s | speak for itself. 1693.52s - 1697.40s | So we submitted this issue in August 2023 1697.40s - 1698.68s | to Metabug Bounty. 1700.04s - 1704.38s | A month later, Meta said there is no update to provide. 1704.38s - 1705.48s | Two months later, 1705.48s - 1708.16s | Meta said they consider the issue mitigated. 1708.16s - 1710.54s | I was suspicious so I went back in 1710.54s - 1713.22s | and just executed some code, 1713.22s - 1715.10s | some shell commands on the runners. 1715.10s - 1716.22s | I didn't do the full attack 1716.22s - 1718.58s | but anyone could still compromise their runners, 1718.58s - 1721.34s | any contributor and based on the workflow logs and stuff, 1721.34s - 1723.34s | I don't think their configurations were still secure 1723.34s - 1724.78s | so we responded saying, 1724.78s - 1727.44s | hey, the issue is not fully mitigated, here's some proof. 1727.44s - 1731.78s | Then another two months went by 1731.78s - 1733.24s | with really no word from Meta 1733.24s - 1736.22s | and we sent a strongly worded email 1736.22s - 1738.54s | expressing concerns on the remediation 1738.54s - 1740.06s | that has been implemented 1740.06s - 1741.18s | which led to some back and forth 1741.18s -


chunk_id: 217_chunk_43
chunk_text:
- 1698.68s | to Metabug Bounty. 1700.04s - 1704.38s | A month later, Meta said there is no update to provide. 1704.38s - 1705.48s | Two months later, 1705.48s - 1708.16s | Meta said they consider the issue mitigated. 1708.16s - 1710.54s | I was suspicious so I went back in 1710.54s - 1713.22s | and just executed some code, 1713.22s - 1715.10s | some shell commands on the runners. 1715.10s - 1716.22s | I didn't do the full attack 1716.22s - 1718.58s | but anyone could still compromise their runners, 1718.58s - 1721.34s | any contributor and based on the workflow logs and stuff, 1721.34s - 1723.34s | I don't think their configurations were still secure 1723.34s - 1724.78s | so we responded saying, 1724.78s - 1727.44s | hey, the issue is not fully mitigated, here's some proof. 1727.44s - 1731.78s | Then another two months went by 1731.78s - 1733.24s | with really no word from Meta 1733.24s - 1736.22s | and we sent a strongly worded email 1736.22s - 1738.54s | expressing concerns on the remediation 1738.54s - 1740.06s | that has been implemented 1740.06s - 1741.18s | which led to some back and forth 1741.18s - 1743.26s | and then December 15th, 1743.26s - 1745.34s | Meta applied some other fixes 1745.34s - 1747.10s | which seemed to be sufficient 1747.10s - 1748.06s | and then they offered a call 1748.06s - 1749.78s | to discuss the remediation in depth. 1749.78s - 1752.14s | And so we actually ended up meeting 1752.14s - 1753.42s | two of the PyTorch maintainers 1753.42s - 1755.80s | and they were very concerned about these issues, 1755.80s - 1757.58s | they were on top of it 1757.58s - 1759.54s | and they also expressed concern 1759.54s - 1761.66s | at the potential gaps in remediation. 1761.66s - 1764.58s | So I'm not totally sure where the disconnect was 1764.58s - 1767.18s | but it worried us that PyTorch 1767.18s - 1771.49s | was potentially vulnerable for this long. 1771.49s - 1773.93s | This attack took a lot of hands-on keyboard, 1773.93s - 1776.49s | workflow modification, command and control, 1776.49s - 1777.33s | all that stuff 1777.33s - 1779.97s | and so it kind of prompted us to say, 1779.97s - 1783.85s | hmm, is there an easier way to automate a lot of this? 1783.85s - 1785.45s | All


chunk_id: 217_chunk_44
chunk_text:
1743.26s | and then December 15th, 1743.26s - 1745.34s | Meta applied some other fixes 1745.34s - 1747.10s | which seemed to be sufficient 1747.10s - 1748.06s | and then they offered a call 1748.06s - 1749.78s | to discuss the remediation in depth. 1749.78s - 1752.14s | And so we actually ended up meeting 1752.14s - 1753.42s | two of the PyTorch maintainers 1753.42s - 1755.80s | and they were very concerned about these issues, 1755.80s - 1757.58s | they were on top of it 1757.58s - 1759.54s | and they also expressed concern 1759.54s - 1761.66s | at the potential gaps in remediation. 1761.66s - 1764.58s | So I'm not totally sure where the disconnect was 1764.58s - 1767.18s | but it worried us that PyTorch 1767.18s - 1771.49s | was potentially vulnerable for this long. 1771.49s - 1773.93s | This attack took a lot of hands-on keyboard, 1773.93s - 1776.49s | workflow modification, command and control, 1776.49s - 1777.33s | all that stuff 1777.33s - 1779.97s | and so it kind of prompted us to say, 1779.97s - 1783.85s | hmm, is there an easier way to automate a lot of this? 1783.85s - 1785.45s | All right, well, now there is. 1785.45s - 1787.85s | So with a tool that I've open sourced this week 1787.85s - 1788.89s | called GatoX, 1788.89s - 1791.09s | it actually automates that self-holstered 1791.09s - 1793.15s | runner takeover process. 1793.15s - 1794.81s | So now instead of all that work 1794.81s - 1796.49s | that you saw John have to do, 1796.49s - 1797.49s | which all you have to do 1797.49s - 1799.21s | is you fix a typo, okay, 1799.21s - 1800.37s | and you become a contributor, 1800.37s - 1801.65s | then you run GatoX 1801.65s - 1803.85s | and then one of two things happens, okay? 1803.85s - 1804.85s | You either get a shell 1804.85s - 1806.93s | and you're on top of the world as hacker man 1806.93s - 1808.83s | or you learn that approval is required 1808.83s - 1811.71s | and then you're sad because you didn't get to hack anything. 1811.71s - 1814.41s | So let's walk through a quick video demo 1814.41s - 1815.81s | of how GatoX works 1815.81s - 1818.77s | to automate this self-holstered runner takeover process. 1818.77s - 1821.45s | So first I use


chunk_id: 217_chunk_45
chunk_text:
right, well, now there is. 1785.45s - 1787.85s | So with a tool that I've open sourced this week 1787.85s - 1788.89s | called GatoX, 1788.89s - 1791.09s | it actually automates that self-holstered 1791.09s - 1793.15s | runner takeover process. 1793.15s - 1794.81s | So now instead of all that work 1794.81s - 1796.49s | that you saw John have to do, 1796.49s - 1797.49s | which all you have to do 1797.49s - 1799.21s | is you fix a typo, okay, 1799.21s - 1800.37s | and you become a contributor, 1800.37s - 1801.65s | then you run GatoX 1801.65s - 1803.85s | and then one of two things happens, okay? 1803.85s - 1804.85s | You either get a shell 1804.85s - 1806.93s | and you're on top of the world as hacker man 1806.93s - 1808.83s | or you learn that approval is required 1808.83s - 1811.71s | and then you're sad because you didn't get to hack anything. 1811.71s - 1814.41s | So let's walk through a quick video demo 1814.41s - 1815.81s | of how GatoX works 1815.81s - 1818.77s | to automate this self-holstered runner takeover process. 1818.77s - 1821.45s | So first I use it to enumerate a runner, okay? 1821.45s - 1822.35s | It's non-ephemeral. 1822.35s - 1824.37s | So now I'm gonna go ahead and fix a typo 1824.37s - 1826.65s | in this test repository here. 1826.65s - 1828.81s | So I go ahead and fix a typo 1828.81s - 1830.09s | and make that commit. 1831.17s - 1832.45s | So after I make the commit, 1832.45s - 1835.41s | I'm going to go ahead and create a pull request. 1835.41s - 1836.97s | So I check that I fixed the typo 1836.97s - 1839.25s | and I make a quick typo fix PR 1839.25s - 1841.01s | and now that's merged in. 1841.01s - 1843.53s | So after that's merged in with my attacker account, 1843.53s - 1845.51s | I'm gonna go ahead and delete the fork. 1845.51s - 1847.49s | So because GatoX will create a new fork 1847.49s - 1850.21s | to throw the payload at the repository. 1850.21s - 1851.89s | So I'm gonna quickly go ahead 1851.89s - 1853.93s | and do the web flow to delete the fork 1853.93s - 1855.49s | and I hit confirm. 1855.49s - 1857.37s | All right, now I'm gonna


chunk_id: 217_chunk_46
chunk_text:
it to enumerate a runner, okay? 1821.45s - 1822.35s | It's non-ephemeral. 1822.35s - 1824.37s | So now I'm gonna go ahead and fix a typo 1824.37s - 1826.65s | in this test repository here. 1826.65s - 1828.81s | So I go ahead and fix a typo 1828.81s - 1830.09s | and make that commit. 1831.17s - 1832.45s | So after I make the commit, 1832.45s - 1835.41s | I'm going to go ahead and create a pull request. 1835.41s - 1836.97s | So I check that I fixed the typo 1836.97s - 1839.25s | and I make a quick typo fix PR 1839.25s - 1841.01s | and now that's merged in. 1841.01s - 1843.53s | So after that's merged in with my attacker account, 1843.53s - 1845.51s | I'm gonna go ahead and delete the fork. 1845.51s - 1847.49s | So because GatoX will create a new fork 1847.49s - 1850.21s | to throw the payload at the repository. 1850.21s - 1851.89s | So I'm gonna quickly go ahead 1851.89s - 1853.93s | and do the web flow to delete the fork 1853.93s - 1855.49s | and I hit confirm. 1855.49s - 1857.37s | All right, now I'm gonna run GatoX. 1857.37s - 1859.21s | So here I've just configured it 1859.21s - 1860.53s | to deploy a runner on runner 1860.53s - 1863.65s | on a Linux x86 64-bit runner. 1863.65s - 1866.11s | So it's gonna go ahead and create the payload 1866.11s - 1868.25s | within a gist and ask for confirmation 1868.25s - 1870.25s | because the next steps are overt. 1870.25s - 1872.57s | So I'm gonna type confirm and hit enter. 1872.57s - 1874.69s | So as soon as I do that, 1874.69s - 1876.43s | it's gonna create a draft pull request 1876.43s - 1878.85s | which is going to deploy that runner on runner payload 1878.85s - 1882.07s | onto that self-holstered runner via fork PR. 1882.07s - 1883.91s | So it's gonna take a second 1883.91s - 1885.45s | because it's now pulling for that runner 1885.45s - 1886.29s | to connect to GitHub 1886.29s - 1888.13s | and then check back into that repository. 1888.13s - 1889.17s | So as soon as it does that, 1889.17s - 1892.93s | it will drop into a nice and very convenient shell. 1892.93s - 1896.25s | Okay, now I'm just gonna test


chunk_id: 217_chunk_47
chunk_text:
run GatoX. 1857.37s - 1859.21s | So here I've just configured it 1859.21s - 1860.53s | to deploy a runner on runner 1860.53s - 1863.65s | on a Linux x86 64-bit runner. 1863.65s - 1866.11s | So it's gonna go ahead and create the payload 1866.11s - 1868.25s | within a gist and ask for confirmation 1868.25s - 1870.25s | because the next steps are overt. 1870.25s - 1872.57s | So I'm gonna type confirm and hit enter. 1872.57s - 1874.69s | So as soon as I do that, 1874.69s - 1876.43s | it's gonna create a draft pull request 1876.43s - 1878.85s | which is going to deploy that runner on runner payload 1878.85s - 1882.07s | onto that self-holstered runner via fork PR. 1882.07s - 1883.91s | So it's gonna take a second 1883.91s - 1885.45s | because it's now pulling for that runner 1885.45s - 1886.29s | to connect to GitHub 1886.29s - 1888.13s | and then check back into that repository. 1888.13s - 1889.17s | So as soon as it does that, 1889.17s - 1892.93s | it will drop into a nice and very convenient shell. 1892.93s - 1896.25s | Okay, now I'm just gonna test this with uname-a 1896.25s - 1897.37s | and when I hit enter, 1897.37s - 1899.13s | what's happening under the hood 1899.13s - 1901.97s | is that GatoX is issuing a workflow dispatch event 1901.97s - 1904.45s | to that workflow in the C2 repository 1904.45s - 1905.89s | which is going to run code 1905.89s - 1908.85s | on that malicious self-holstered runner 1908.85s - 1910.79s | and you can see that output there. 1910.79s - 1913.33s | All right, so now you know the techniques 1913.33s - 1915.81s | and now since I've open sourced it, you have a tool. 1915.83s - 1917.05s | And another thing I wanna point out 1917.05s - 1920.13s | is that finding CICD misconfigurations 1920.13s - 1923.01s | in open source repositories can be a very thankless job 1923.01s - 1925.29s | because you submit a report, they fix a bug 1925.29s - 1926.49s | and you don't even get a CVE. 1926.49s - 1928.25s | So there's not really a lot of motivation for people. 1928.25s - 1931.69s | So what I've done with GatoX is that on its wiki, 1931.69s - 1933.01s | I've created a little hall of fame.


chunk_id: 217_chunk_48
chunk_text:
this with uname-a 1896.25s - 1897.37s | and when I hit enter, 1897.37s - 1899.13s | what's happening under the hood 1899.13s - 1901.97s | is that GatoX is issuing a workflow dispatch event 1901.97s - 1904.45s | to that workflow in the C2 repository 1904.45s - 1905.89s | which is going to run code 1905.89s - 1908.85s | on that malicious self-holstered runner 1908.85s - 1910.79s | and you can see that output there. 1910.79s - 1913.33s | All right, so now you know the techniques 1913.33s - 1915.81s | and now since I've open sourced it, you have a tool. 1915.83s - 1917.05s | And another thing I wanna point out 1917.05s - 1920.13s | is that finding CICD misconfigurations 1920.13s - 1923.01s | in open source repositories can be a very thankless job 1923.01s - 1925.29s | because you submit a report, they fix a bug 1925.29s - 1926.49s | and you don't even get a CVE. 1926.49s - 1928.25s | So there's not really a lot of motivation for people. 1928.25s - 1931.69s | So what I've done with GatoX is that on its wiki, 1931.69s - 1933.01s | I've created a little hall of fame. 1933.01s - 1935.73s | So if you use GatoX to find a pwn request, 1935.73s - 1937.85s | injection or self-holstered runner vulnerability 1937.85s - 1940.71s | in open source project and the maintainers fix it, 1940.71s - 1941.55s | then just make an issue 1941.55s - 1944.17s | and I'll add your name to the wiki to get some credit. 1944.17s - 1945.69s | Just make sure it's actually fixed 1946.53s - 1947.37s | because we don't want the hall of fame 1947.37s - 1948.19s | to be the hall of O days 1948.19s - 1951.64s | as much as some people might like that. 1951.64s - 1954.20s | Okay, so we showed all of the technical TTPs 1954.20s - 1956.48s | we use during our PyTorch attack, 1956.48s - 1957.92s | but there are a bunch of other stuff 1957.92s - 1959.68s | we've used on other targets. 1959.68s - 1962.44s | And so we wanna take these next few minutes 1962.44s - 1964.84s | to honor the diehard CICD nerds 1964.84s - 1967.28s | who are here to learn all of the TTPs possible. 1967.28s - 1969.48s | So we're gonna run through this pretty quickly, 1969.48s


chunk_id: 217_chunk_49
chunk_text:
1933.01s - 1935.73s | So if you use GatoX to find a pwn request, 1935.73s - 1937.85s | injection or self-holstered runner vulnerability 1937.85s - 1940.71s | in open source project and the maintainers fix it, 1940.71s - 1941.55s | then just make an issue 1941.55s - 1944.17s | and I'll add your name to the wiki to get some credit. 1944.17s - 1945.69s | Just make sure it's actually fixed 1946.53s - 1947.37s | because we don't want the hall of fame 1947.37s - 1948.19s | to be the hall of O days 1948.19s - 1951.64s | as much as some people might like that. 1951.64s - 1954.20s | Okay, so we showed all of the technical TTPs 1954.20s - 1956.48s | we use during our PyTorch attack, 1956.48s - 1957.92s | but there are a bunch of other stuff 1957.92s - 1959.68s | we've used on other targets. 1959.68s - 1962.44s | And so we wanna take these next few minutes 1962.44s - 1964.84s | to honor the diehard CICD nerds 1964.84s - 1967.28s | who are here to learn all of the TTPs possible. 1967.28s - 1969.48s | So we're gonna run through this pretty quickly, 1969.48s - 1971.08s | but all the slides will be up there online 1971.08s - 1972.72s | so you can check it out after. 1972.72s - 1974.32s | All right, so one thing you can do 1974.32s - 1975.68s | after getting persistence on a runner 1975.68s - 1977.92s | is a solo run style build compromise. 1977.92s - 1980.52s | So you get the persistence 1980.52s - 1983.66s | and then you just have to modify scripts 1983.66s - 1986.22s | for the build or source code after it's checked out. 1986.22s - 1987.28s | And once you do that, 1987.28s - 1990.16s | the final build artifacts from that run 1990.16s - 1991.30s | are going to be poisoned, 1991.30s - 1993.08s | but it's not gonna be linked back 1993.08s - 1994.04s | to the original source code. 1994.04s - 1995.36s | So it's a very stealthy way 1995.36s - 1997.76s | to conduct a supply chain attack. 1997.76s - 2000.18s | And here's another way that you could take 2000.18s - 2002.48s | GitHub release asset tampering a little further. 2002.48s - 2004.72s | So if you have that context write token, 2004.72s - 2007.68s | then


chunk_id: 217_chunk_50
chunk_text:
- 1971.08s | but all the slides will be up there online 1971.08s - 1972.72s | so you can check it out after. 1972.72s - 1974.32s | All right, so one thing you can do 1974.32s - 1975.68s | after getting persistence on a runner 1975.68s - 1977.92s | is a solo run style build compromise. 1977.92s - 1980.52s | So you get the persistence 1980.52s - 1983.66s | and then you just have to modify scripts 1983.66s - 1986.22s | for the build or source code after it's checked out. 1986.22s - 1987.28s | And once you do that, 1987.28s - 1990.16s | the final build artifacts from that run 1990.16s - 1991.30s | are going to be poisoned, 1991.30s - 1993.08s | but it's not gonna be linked back 1993.08s - 1994.04s | to the original source code. 1994.04s - 1995.36s | So it's a very stealthy way 1995.36s - 1997.76s | to conduct a supply chain attack. 1997.76s - 2000.18s | And here's another way that you could take 2000.18s - 2002.48s | GitHub release asset tampering a little further. 2002.48s - 2004.72s | So if you have that context write token, 2004.72s - 2007.68s | then all you have to do to tamper with releases on assets 2007.68s - 2010.12s | is use the API to delete the old asset 2010.12s - 2011.92s | and then upload a new one with a post request 2011.92s - 2013.16s | right on top of it. 2013.16s - 2015.76s | And the indicator of compromise here 2015.76s - 2017.68s | is just going to be the timestamp 2017.68s - 2020.80s | when a user is looking at it through the web interface. 2020.80s - 2022.34s | So it's not very obvious. 2022.34s - 2025.24s | And if you wanna see a tale of a bug bounty gone wrong 2025.24s - 2027.36s | and a program acting in some pretty bad faith, 2027.36s - 2030.36s | go check out my blog post on my report to the ASTAR network, 2030.36s - 2040.17s | which is a tiny polka dot parachain. 2040.17s - 2042.93s | All right, another technique you can add to your arsenal 2042.93s - 2045.17s | is something called a post checkout hook. 2045.17s - 2049.01s | So what happens when a subsequent workflow 2049.01s - 2051.25s | that you need to get secrets from or token from


chunk_id: 217_chunk_51
chunk_text:
all you have to do to tamper with releases on assets 2007.68s - 2010.12s | is use the API to delete the old asset 2010.12s - 2011.92s | and then upload a new one with a post request 2011.92s - 2013.16s | right on top of it. 2013.16s - 2015.76s | And the indicator of compromise here 2015.76s - 2017.68s | is just going to be the timestamp 2017.68s - 2020.80s | when a user is looking at it through the web interface. 2020.80s - 2022.34s | So it's not very obvious. 2022.34s - 2025.24s | And if you wanna see a tale of a bug bounty gone wrong 2025.24s - 2027.36s | and a program acting in some pretty bad faith, 2027.36s - 2030.36s | go check out my blog post on my report to the ASTAR network, 2030.36s - 2040.17s | which is a tiny polka dot parachain. 2040.17s - 2042.93s | All right, another technique you can add to your arsenal 2042.93s - 2045.17s | is something called a post checkout hook. 2045.17s - 2049.01s | So what happens when a subsequent workflow 2049.01s - 2051.25s | that you need to get secrets from or token from 2051.25s - 2053.81s | runs only very infrequently? 2053.81s - 2056.69s | Well, you need something that can have these qualities. 2056.69s - 2059.11s | It needs, you need to be able to extend the build time. 2059.11s - 2061.51s | You also don't wanna break how the workflow works 2061.51s - 2063.79s | to inform someone that something's going wrong. 2063.79s - 2065.13s | And you also wanna get notified 2065.13s - 2067.65s | because you don't wanna be pressing F5 for two weeks. 2067.69s - 2071.37s | Well, a quick solution here is to place a script like this 2071.37s - 2073.93s | into the git hooks directory of the repository 2073.93s - 2075.73s | in the original self-hosted runner. 2075.73s - 2079.29s | This example here will just take that git config file 2079.29s - 2081.89s | and code it and then send it off to your XFIL URL 2081.89s - 2083.25s | and then sleep for 15 minutes. 2083.25s - 2084.93s | So this is gonna give you 15 minutes 2084.93s - 2086.57s | to take advantage of that GitHub token 2086.57s - 2089.57s | to perform post exploitation actions. 2089.57s - 2092.61s | All right,


chunk_id: 217_chunk_52
chunk_text:
2051.25s - 2053.81s | runs only very infrequently? 2053.81s - 2056.69s | Well, you need something that can have these qualities. 2056.69s - 2059.11s | It needs, you need to be able to extend the build time. 2059.11s - 2061.51s | You also don't wanna break how the workflow works 2061.51s - 2063.79s | to inform someone that something's going wrong. 2063.79s - 2065.13s | And you also wanna get notified 2065.13s - 2067.65s | because you don't wanna be pressing F5 for two weeks. 2067.69s - 2071.37s | Well, a quick solution here is to place a script like this 2071.37s - 2073.93s | into the git hooks directory of the repository 2073.93s - 2075.73s | in the original self-hosted runner. 2075.73s - 2079.29s | This example here will just take that git config file 2079.29s - 2081.89s | and code it and then send it off to your XFIL URL 2081.89s - 2083.25s | and then sleep for 15 minutes. 2083.25s - 2084.93s | So this is gonna give you 15 minutes 2084.93s - 2086.57s | to take advantage of that GitHub token 2086.57s - 2089.57s | to perform post exploitation actions. 2089.57s - 2092.61s | All right, before we dive into other TTPs, 2092.61s - 2095.17s | it's important to remember how self-hosted runner takeover 2095.17s - 2096.61s | falls in within the landscape 2096.61s - 2099.23s | of broader GitHub actions attack techniques. 2099.23s - 2101.97s | So there's another type of attack called bone requests 2101.97s - 2105.41s | where you basically get execution 2105.41s - 2107.01s | within a privileged workflow. 2107.01s - 2108.93s | That's the same thing that happens with runner takeover. 2108.93s - 2111.05s | From there, there's different 2111.05s - 2112.61s | post exploitation techniques you can use. 2112.61s - 2114.25s | So you can steal secrets, 2114.25s - 2116.25s | you could get the actions runtime token, 2116.25s - 2118.81s | which has some interesting attacks with cache poisoning, 2118.81s - 2120.57s | as well as OIDC abuse, 2120.57s - 2123.65s | which I think the talk right after us in this track 2123.65s - 2125.09s | actually goes into a lot more detail there. 2125.93s - 2126.77s | So kinda cool. 2126.77s - 2129.09s | And then there's just too many different attacks 2129.09s - 2129.93s | to talk about. 2129.93s - 2132.35s | There's so many ways


chunk_id: 217_chunk_53
chunk_text:
before we dive into other TTPs, 2092.61s - 2095.17s | it's important to remember how self-hosted runner takeover 2095.17s - 2096.61s | falls in within the landscape 2096.61s - 2099.23s | of broader GitHub actions attack techniques. 2099.23s - 2101.97s | So there's another type of attack called bone requests 2101.97s - 2105.41s | where you basically get execution 2105.41s - 2107.01s | within a privileged workflow. 2107.01s - 2108.93s | That's the same thing that happens with runner takeover. 2108.93s - 2111.05s | From there, there's different 2111.05s - 2112.61s | post exploitation techniques you can use. 2112.61s - 2114.25s | So you can steal secrets, 2114.25s - 2116.25s | you could get the actions runtime token, 2116.25s - 2118.81s | which has some interesting attacks with cache poisoning, 2118.81s - 2120.57s | as well as OIDC abuse, 2120.57s - 2123.65s | which I think the talk right after us in this track 2123.65s - 2125.09s | actually goes into a lot more detail there. 2125.93s - 2126.77s | So kinda cool. 2126.77s - 2129.09s | And then there's just too many different attacks 2129.09s - 2129.93s | to talk about. 2129.93s - 2132.35s | There's so many ways to achieve great impact. 2133.25s - 2134.73s | We talked about some of the permissions 2134.73s - 2135.77s | of the GitHub token today, 2135.77s - 2138.09s | but there's a lot of other dangerous permissions out there. 2138.09s - 2139.81s | So some of them don't pose a risk, 2139.81s - 2141.93s | but others like contents right and actions right 2141.93s - 2143.93s | obviously pose a significant risk. 2143.93s - 2146.13s | So other permissions we have identified 2146.13s - 2148.37s | as dangerous and actually abused during our attacks 2148.37s - 2151.45s | are pages right, pull requests right, and packages right. 2151.45s - 2152.89s | And there's a lot of good documentation online 2152.89s - 2156.17s | if you wanna learn more about what these actually do. 2156.17s - 2158.13s | If you have just actions right and contents right, 2158.13s - 2159.81s | you can create feature branches, 2159.81s - 2161.69s | modify files, execute in workflows, 2161.69s - 2164.33s | and then issue dispatches events to steal secrets. 2164.33s - 2165.25s | If that looks familiar, 2165.25s - 2168.33s | it's because that's exactly what we did during PyTorch. 2168.33s - 2170.65s |


