# MediaChunks

chunk_text:
the money actually shows up, this is just a weird government 1264.86s - 1267.42s | thing because money is weird in the government. 1267.42s - 1271.10s | Like, so the money shows up, and you have three months to spend it because you need 1271.10s - 1274.22s | to spend it three months in advance of the end of the fiscal year, and it's a nightmare 1275.42s - 1277.58s | for so many reasons that you can probably figure out on your own. 1279.34s - 1282.14s | Time is valuable, I think we've got to start speed running this. 1282.86s - 1285.74s | Time is valuable, yes, we're going to move on. 1286.70s - 1292.38s | It took three months to get a three-month contract signed and started, that's the story 1292.38s - 1293.26s | I have there, it was awful. 1294.14s - 1297.66s | Tech talent is expensive, it's real hard to get people to go from making a ton of money 1297.66s - 1301.58s | at a Google or an Amazon and ask them to take a pay cut to come do civil service and work 1301.58s - 1305.26s | for the DOD, they just can't compete. 1305.26s - 1309.66s | Selling to the government is absolutely a pain, you know, there's a lot of requirements 1309.66s - 1314.06s | that you have to fill out, it's kind of like a decision you have to make as a company where 1314.06s - 1319.26s | once you've kind of had a bigger adoption from the consumer world, and you're like, 1319.26s - 1322.54s | okay, we can turn to the government, and then someone tells you that, oh, you can sell the 1322.54s - 1325.82s | same thing to the government six different times, you can make a lot of money doing that, 1325.82s - 1327.66s | and then you kind of prioritize that, right? 1328.94s - 1333.26s | So some of the suggestions we have, we want to be able to create incentives for technical 1333.26s - 1334.54s | experts to join and stay. 1334.54s - 1338.78s | And the answer there is not necessarily more money, it's probably one of the most trite 1338.78s - 1342.14s | examples I've heard people say, oh, we can't pay enough, so we're never going to succeed, 1343.02s - 1346.94s | especially once you've successfully recruited someone, you're already paying enough,


chunk_text:
- 1309.66s | Selling to the government is absolutely a pain, you know, there's a lot of requirements 1309.66s - 1314.06s | that you have to fill out, it's kind of like a decision you have to make as a company where 1314.06s - 1319.26s | once you've kind of had a bigger adoption from the consumer world, and you're like, 1319.26s - 1322.54s | okay, we can turn to the government, and then someone tells you that, oh, you can sell the 1322.54s - 1325.82s | same thing to the government six different times, you can make a lot of money doing that, 1325.82s - 1327.66s | and then you kind of prioritize that, right? 1328.94s - 1333.26s | So some of the suggestions we have, we want to be able to create incentives for technical 1333.26s - 1334.54s | experts to join and stay. 1334.54s - 1338.78s | And the answer there is not necessarily more money, it's probably one of the most trite 1338.78s - 1342.14s | examples I've heard people say, oh, we can't pay enough, so we're never going to succeed, 1343.02s - 1346.94s | especially once you've successfully recruited someone, you're already paying enough, so 1346.94s - 1350.22s | then there's other ways to get them to stay, and it's not usually money, I'm not saying 1350.22s - 1353.98s | money doesn't help, and some organizations like CISA, for example, have done incredible 1353.98s - 1358.46s | things at raising the actual pay for technical contributors, and also the 90th has done a 1358.46s - 1359.02s | great job too. 1359.02s - 1363.66s | Yeah, this use it or lose it financial policy is really frustrating to hear, that if you 1363.66s - 1367.74s | don't spend all of your money, you don't get the same amount or more next year, so you're 1367.74s - 1371.74s | incentivized to just blow money towards the end of the year, so you can say, oh, we spent 1371.74s - 1374.70s | it all, we need more money next year, and that's ridiculous, and that's not just a thing 1374.70s - 1378.54s | for the government, that happens in corporate too, but those are policies that absolutely 1378.54s - 1379.98s | do not lead to the right incentives. 1380.78s - 1384.86s | The next one is delete, delete, delete, so the idea of as you are creating


chunk_text:
so 1346.94s - 1350.22s | then there's other ways to get them to stay, and it's not usually money, I'm not saying 1350.22s - 1353.98s | money doesn't help, and some organizations like CISA, for example, have done incredible 1353.98s - 1358.46s | things at raising the actual pay for technical contributors, and also the 90th has done a 1358.46s - 1359.02s | great job too. 1359.02s - 1363.66s | Yeah, this use it or lose it financial policy is really frustrating to hear, that if you 1363.66s - 1367.74s | don't spend all of your money, you don't get the same amount or more next year, so you're 1367.74s - 1371.74s | incentivized to just blow money towards the end of the year, so you can say, oh, we spent 1371.74s - 1374.70s | it all, we need more money next year, and that's ridiculous, and that's not just a thing 1374.70s - 1378.54s | for the government, that happens in corporate too, but those are policies that absolutely 1378.54s - 1379.98s | do not lead to the right incentives. 1380.78s - 1384.86s | The next one is delete, delete, delete, so the idea of as you are creating more things, 1384.86s - 1389.50s | you also need to be uncreating other things, if you set up a task force to make sure that 1389.50s - 1393.10s | all of the asbestos was out of the building, you don't need to keep the task force when 1393.10s - 1394.78s | the asbestos is out of the building. 1394.78s - 1397.58s | Yeah, and then prioritize quality over quantity. 1399.18s - 1403.66s | One staff engineer can't be replaced with ten junior engineers, it doesn't work that 1403.66s - 1407.74s | way, right, you need to bring in technical talent and expertise from a leadership level 1407.74s - 1411.58s | and keep those folks around, and you need to prioritize having that quality. 1413.58s - 1418.30s | All right, so the next one, private partnerships are challenging. 1418.30s - 1420.62s | Public private partnerships are challenging, there we go. 1421.74s - 1426.46s | So we were at this conference, and this colonel I was talking to, he said he led some research 1426.46s - 1430.14s | and training facility for the Air Force and how they taught people how to come in and 1430.14s - 1434.54s | learn how to


chunk_text:
more things, 1384.86s - 1389.50s | you also need to be uncreating other things, if you set up a task force to make sure that 1389.50s - 1393.10s | all of the asbestos was out of the building, you don't need to keep the task force when 1393.10s - 1394.78s | the asbestos is out of the building. 1394.78s - 1397.58s | Yeah, and then prioritize quality over quantity. 1399.18s - 1403.66s | One staff engineer can't be replaced with ten junior engineers, it doesn't work that 1403.66s - 1407.74s | way, right, you need to bring in technical talent and expertise from a leadership level 1407.74s - 1411.58s | and keep those folks around, and you need to prioritize having that quality. 1413.58s - 1418.30s | All right, so the next one, private partnerships are challenging. 1418.30s - 1420.62s | Public private partnerships are challenging, there we go. 1421.74s - 1426.46s | So we were at this conference, and this colonel I was talking to, he said he led some research 1426.46s - 1430.14s | and training facility for the Air Force and how they taught people how to come in and 1430.14s - 1434.54s | learn how to code and use Kubernetes and some other modern stuff, and my face lit up, I'm 1434.54s - 1438.30s | like, cool, I'd love to come talk to your students and teach them about some of the 1438.30s - 1441.26s | cool stuff we're working in Kubernetes, and the guy stonewalled me and didn't want to 1441.26s - 1444.70s | talk to me because he thought I was trying to sell him services and stuff. 1444.70s - 1448.86s | And I was just genuinely excited as a lead for an open source project to come talk about 1448.86s - 1452.62s | the cool things that we were building, and so once Rebecca actually explained that to 1452.62s - 1455.74s | him, his whole demeanor changed and he was much more open to a conversation. 1457.58s - 1460.86s | This one's funny, so if you've ever worked in the government, you show up to a conference 1460.86s - 1465.26s | and you ask for your landing fee because you're going to need to have snacks and somebody 1465.26s - 1466.14s | went and bought those. 1466.78s - 1470.78s | So we get to the Bravo hackathon and we're asked for a landing


chunk_text:
code and use Kubernetes and some other modern stuff, and my face lit up, I'm 1434.54s - 1438.30s | like, cool, I'd love to come talk to your students and teach them about some of the 1438.30s - 1441.26s | cool stuff we're working in Kubernetes, and the guy stonewalled me and didn't want to 1441.26s - 1444.70s | talk to me because he thought I was trying to sell him services and stuff. 1444.70s - 1448.86s | And I was just genuinely excited as a lead for an open source project to come talk about 1448.86s - 1452.62s | the cool things that we were building, and so once Rebecca actually explained that to 1452.62s - 1455.74s | him, his whole demeanor changed and he was much more open to a conversation. 1457.58s - 1460.86s | This one's funny, so if you've ever worked in the government, you show up to a conference 1460.86s - 1465.26s | and you ask for your landing fee because you're going to need to have snacks and somebody 1465.26s - 1466.14s | went and bought those. 1466.78s - 1470.78s | So we get to the Bravo hackathon and we're asked for a landing fee and Eddie says, what 1470.78s - 1472.78s | the fuck is a landing fee? 1472.78s - 1475.66s | I'm like, well, it's for the snacks, and to me, again, this is one of those things that's 1475.66s - 1479.10s | so obvious, he says, well, why didn't someone just go buy the snacks? 1479.10s - 1482.94s | Well, somebody did, we've got to pay them back, and so then he says, well, can't Defense 1482.94s - 1484.94s | Unicorns, our company, just sponsor them? 1484.94s - 1488.30s | I can just go swipe my company card and buy some snacks and then we don't have to do this 1488.30s - 1494.06s | landing fee garbage, and I was like, so we could, but the paperwork that would be required 1494.06s - 1499.58s | by the government to accept our donation of snacks would not be worth it. 1499.58s - 1503.18s | This is simpler, I promise, just pay 20 bucks, have some Cheetos. 1505.02s - 1506.14s | So what makes it hard? 1506.14s - 1510.86s | Ethics requirements, that whole thing of like, look, if you look at how Russia and China 1510.86s - 1516.78s | do business when


chunk_text:
fee and Eddie says, what 1470.78s - 1472.78s | the fuck is a landing fee? 1472.78s - 1475.66s | I'm like, well, it's for the snacks, and to me, again, this is one of those things that's 1475.66s - 1479.10s | so obvious, he says, well, why didn't someone just go buy the snacks? 1479.10s - 1482.94s | Well, somebody did, we've got to pay them back, and so then he says, well, can't Defense 1482.94s - 1484.94s | Unicorns, our company, just sponsor them? 1484.94s - 1488.30s | I can just go swipe my company card and buy some snacks and then we don't have to do this 1488.30s - 1494.06s | landing fee garbage, and I was like, so we could, but the paperwork that would be required 1494.06s - 1499.58s | by the government to accept our donation of snacks would not be worth it. 1499.58s - 1503.18s | This is simpler, I promise, just pay 20 bucks, have some Cheetos. 1505.02s - 1506.14s | So what makes it hard? 1506.14s - 1510.86s | Ethics requirements, that whole thing of like, look, if you look at how Russia and China 1510.86s - 1516.78s | do business when it comes to contracting, there is a favored set of contractors who 1516.78s - 1523.74s | get all of the work from food service to, I don't know, running newspapers, and that 1523.74s - 1526.86s | is not how this country should be, and we don't want it to be. 1526.86s - 1531.74s | So instead, we put together these really complicated, complex ethics requirements that make it almost 1531.74s - 1534.06s | impossible to even have lunch with your friend who's a contractor. 1534.70s - 1538.06s | I'm not saying that's necessarily the right answer either, and there's probably somewhere 1538.06s - 1539.42s | in between we need to be. 1539.42s - 1543.50s | They also tend to make people really, really gun-shy about things that they're not sure 1543.50s - 1547.58s | whether or not are an ethical issue, and so they'll just default to like, hey, I'm not 1547.58s - 1552.14s | going to talk to you, look at you, or ever speak to you again, Rebecca, now that you've 1552.14s - 1554.22s | left the government because pretty much you're dead to me. 1555.26s - 1557.50s | Next is accidentally screwing up active acquisitions. 1557.50s -


chunk_text:
it comes to contracting, there is a favored set of contractors who 1516.78s - 1523.74s | get all of the work from food service to, I don't know, running newspapers, and that 1523.74s - 1526.86s | is not how this country should be, and we don't want it to be. 1526.86s - 1531.74s | So instead, we put together these really complicated, complex ethics requirements that make it almost 1531.74s - 1534.06s | impossible to even have lunch with your friend who's a contractor. 1534.70s - 1538.06s | I'm not saying that's necessarily the right answer either, and there's probably somewhere 1538.06s - 1539.42s | in between we need to be. 1539.42s - 1543.50s | They also tend to make people really, really gun-shy about things that they're not sure 1543.50s - 1547.58s | whether or not are an ethical issue, and so they'll just default to like, hey, I'm not 1547.58s - 1552.14s | going to talk to you, look at you, or ever speak to you again, Rebecca, now that you've 1552.14s - 1554.22s | left the government because pretty much you're dead to me. 1555.26s - 1557.50s | Next is accidentally screwing up active acquisitions. 1557.50s - 1559.18s | They take a really freaking long time. 1559.18s - 1565.10s | If you screw it up and have to start over, that sucks, and so if you have an active acquisition 1565.74s - 1569.58s | in the pipeline, you're not going to speak to anyone about it because you're terrified, 1569.58s - 1571.34s | and I almost understand that. 1571.34s - 1572.86s | The next one, stop trying to sell me stuff. 1573.42s - 1578.22s | When I was in the government, I had an email black box that as soon as you as a contractor 1578.22s - 1581.90s | sent me a single email that was anything other than like directly related to current 1581.90s - 1585.18s | existing work, you went to the black box, and I never saw your emails again or thought 1585.18s - 1586.30s | about you ever. 1587.10s - 1593.58s | This was not probably the best choice in retrospect, but also like Eddie absolutely would have 1593.58s - 1597.50s | been in my black box the minute he was like, hey, I want to tell your people about some 1597.50s - 1597.82s | stuff. 1597.82s - 1598.94s | Can I come


chunk_text:
1559.18s | They take a really freaking long time. 1559.18s - 1565.10s | If you screw it up and have to start over, that sucks, and so if you have an active acquisition 1565.74s - 1569.58s | in the pipeline, you're not going to speak to anyone about it because you're terrified, 1569.58s - 1571.34s | and I almost understand that. 1571.34s - 1572.86s | The next one, stop trying to sell me stuff. 1573.42s - 1578.22s | When I was in the government, I had an email black box that as soon as you as a contractor 1578.22s - 1581.90s | sent me a single email that was anything other than like directly related to current 1581.90s - 1585.18s | existing work, you went to the black box, and I never saw your emails again or thought 1585.18s - 1586.30s | about you ever. 1587.10s - 1593.58s | This was not probably the best choice in retrospect, but also like Eddie absolutely would have 1593.58s - 1597.50s | been in my black box the minute he was like, hey, I want to tell your people about some 1597.50s - 1597.82s | stuff. 1597.82s - 1598.94s | Can I come eat a lunch and learn? 1598.94s - 1600.70s | I'd be like, hell no, and also black box. 1601.50s - 1604.86s | The last one is just this unwillingness to articulate what it is that you need, this 1604.94s - 1608.46s | I'll know it when I see it mentality that I've seen in a lot of government employees, 1608.46s - 1612.54s | which like I'm not going to even collaborate with you on developing a requirement. 1612.54s - 1616.30s | I'm just going to wait until something comes along and like a beacon of light, it inspires 1616.30s - 1620.46s | me, and that's what I'm going to buy, and when I'm trying to decide how to spend internal 1620.46s - 1625.02s | research and development funds, that is not going to be an effective mechanism for prioritizing 1625.02s - 1625.18s | them. 1625.98s - 1626.22s | Yeah. 1627.34s - 1629.90s | As an open source lead, we have different foundations. 1629.90s - 1633.90s | We have so many working groups with empty seats at the table, and government folks, 1633.90s - 1635.66s | we want you to come join us. 1635.66s - 1636.78s | We don't know how


chunk_text:
eat a lunch and learn? 1598.94s - 1600.70s | I'd be like, hell no, and also black box. 1601.50s - 1604.86s | The last one is just this unwillingness to articulate what it is that you need, this 1604.94s - 1608.46s | I'll know it when I see it mentality that I've seen in a lot of government employees, 1608.46s - 1612.54s | which like I'm not going to even collaborate with you on developing a requirement. 1612.54s - 1616.30s | I'm just going to wait until something comes along and like a beacon of light, it inspires 1616.30s - 1620.46s | me, and that's what I'm going to buy, and when I'm trying to decide how to spend internal 1620.46s - 1625.02s | research and development funds, that is not going to be an effective mechanism for prioritizing 1625.02s - 1625.18s | them. 1625.98s - 1626.22s | Yeah. 1627.34s - 1629.90s | As an open source lead, we have different foundations. 1629.90s - 1633.90s | We have so many working groups with empty seats at the table, and government folks, 1633.90s - 1635.66s | we want you to come join us. 1635.66s - 1636.78s | We don't know how to reach out to you. 1636.78s - 1640.14s | We don't know how to invite you to these working groups, but there's too many empty seats that 1640.14s - 1641.98s | we need filled from people in the public sector. 1642.54s - 1644.94s | We can't solve your problems if we don't know about them. 1644.94s - 1649.18s | Someone told me the other day, I work on kubectl, a command line tool for Kubernetes, and they 1649.18s - 1653.74s | said if you made this one small change, it would make my day-to-day life 100 times better, 1653.74s - 1657.98s | and I was like, cool, let's go fix that right now, and their mind was blown that that was 1657.98s - 1660.62s | just something like you can file these feature requests. 1660.62s - 1664.30s | This is something you're allowed to do, but we don't know what your priorities are, and 1664.30s - 1666.30s | we can't prioritize this work unless you tell us. 1667.34s - 1670.86s | There's this stigma in the open source community, too, that a lot of folks that are building 1670.86s - 1674.06s | open source software don't want their software used


chunk_text:
to reach out to you. 1636.78s - 1640.14s | We don't know how to invite you to these working groups, but there's too many empty seats that 1640.14s - 1641.98s | we need filled from people in the public sector. 1642.54s - 1644.94s | We can't solve your problems if we don't know about them. 1644.94s - 1649.18s | Someone told me the other day, I work on kubectl, a command line tool for Kubernetes, and they 1649.18s - 1653.74s | said if you made this one small change, it would make my day-to-day life 100 times better, 1653.74s - 1657.98s | and I was like, cool, let's go fix that right now, and their mind was blown that that was 1657.98s - 1660.62s | just something like you can file these feature requests. 1660.62s - 1664.30s | This is something you're allowed to do, but we don't know what your priorities are, and 1664.30s - 1666.30s | we can't prioritize this work unless you tell us. 1667.34s - 1670.86s | There's this stigma in the open source community, too, that a lot of folks that are building 1670.86s - 1674.06s | open source software don't want their software used by the military. 1675.02s - 1676.14s | It's kind of a double-edged sword. 1676.14s - 1680.70s | That's the beauty of open source is open source allows anyone to take, modify, and use that 1680.70s - 1686.78s | software, so it's kind of a balance we need to work through, but it's, you know, the idea 1686.78s - 1691.10s | that the software is open source, we don't want certain groups to use it, it doesn't 1691.10s - 1695.82s | meet the spirit of open source, and I legit just want to help, just like with that kernel. 1695.82s - 1699.42s | A lot of open source maintainers, we're building cool things that we care about and are super 1699.42s - 1700.30s | passionate about. 1700.30s - 1701.50s | We really want to help. 1701.50s - 1703.26s | We don't care about selling you things. 1703.26s - 1707.02s | Hopefully a lot of us that work on large projects are funded by an employer to do this work 1707.02s - 1709.42s | in open source, so engage with us. 1709.42s - 1710.54s | Make feature requests. 1710.54s - 1711.66s | That is what we want you to do. 1713.02s -


chunk_text:
by the military. 1675.02s - 1676.14s | It's kind of a double-edged sword. 1676.14s - 1680.70s | That's the beauty of open source is open source allows anyone to take, modify, and use that 1680.70s - 1686.78s | software, so it's kind of a balance we need to work through, but it's, you know, the idea 1686.78s - 1691.10s | that the software is open source, we don't want certain groups to use it, it doesn't 1691.10s - 1695.82s | meet the spirit of open source, and I legit just want to help, just like with that kernel. 1695.82s - 1699.42s | A lot of open source maintainers, we're building cool things that we care about and are super 1699.42s - 1700.30s | passionate about. 1700.30s - 1701.50s | We really want to help. 1701.50s - 1703.26s | We don't care about selling you things. 1703.26s - 1707.02s | Hopefully a lot of us that work on large projects are funded by an employer to do this work 1707.02s - 1709.42s | in open source, so engage with us. 1709.42s - 1710.54s | Make feature requests. 1710.54s - 1711.66s | That is what we want you to do. 1713.02s - 1714.06s | How do we fix it? 1714.06s - 1715.02s | I don't know. 1715.02s - 1715.82s | I'm done. 1715.82s - 1716.22s | Just kidding. 1717.10s - 1721.34s | One of the things actually I love are these no-cost agreements called Cooperative Research 1721.34s - 1722.46s | and Development Agreements. 1722.46s - 1728.06s | They let the government share information and vice versa with academia, industry, and 1728.06s - 1731.18s | it's a really great opportunity to, one, make it clear you're not trying to sell something, 1731.18s - 1735.18s | you're legit trying to collaborate, but also kind of break down some of the barriers that 1735.18s - 1738.86s | would make you maybe more resistant to sharing information within the government. 1738.86s - 1742.46s | Yeah, and one of the things I've learned about this is that there's kind of two steps to 1742.46s - 1745.58s | building an implementation for a government solution. 1745.58s - 1750.94s | There's the design phase, the proposal piece, and then there's the implementation piece, 1750.94s - 1755.34s | and if you do the first one, you're ineligible to bid and do the second one, so obviously 1755.34s - 1759.58s |


chunk_text:
1714.06s | How do we fix it? 1714.06s - 1715.02s | I don't know. 1715.02s - 1715.82s | I'm done. 1715.82s - 1716.22s | Just kidding. 1717.10s - 1721.34s | One of the things actually I love are these no-cost agreements called Cooperative Research 1721.34s - 1722.46s | and Development Agreements. 1722.46s - 1728.06s | They let the government share information and vice versa with academia, industry, and 1728.06s - 1731.18s | it's a really great opportunity to, one, make it clear you're not trying to sell something, 1731.18s - 1735.18s | you're legit trying to collaborate, but also kind of break down some of the barriers that 1735.18s - 1738.86s | would make you maybe more resistant to sharing information within the government. 1738.86s - 1742.46s | Yeah, and one of the things I've learned about this is that there's kind of two steps to 1742.46s - 1745.58s | building an implementation for a government solution. 1745.58s - 1750.94s | There's the design phase, the proposal piece, and then there's the implementation piece, 1750.94s - 1755.34s | and if you do the first one, you're ineligible to bid and do the second one, so obviously 1755.34s - 1759.58s | there's more money and ongoing support fees in the second one, so most groups want to 1759.58s - 1763.66s | do that type of work and not the first group, so what you wind up with is people who are 1763.66s - 1769.82s | technically competent implementing and building things that might not be designed the best 1769.82s - 1771.58s | possible way, and that's not good either. 1772.54s - 1775.42s | I think it's just finding ways to engage and learn from industry and the tech community, 1775.42s - 1778.86s | so if you are in this room, you are doing that because you showed up, if you're a government 1778.86s - 1781.74s | person in this room anyway, you showed up to DEF CON, so good start. 1781.74s - 1785.74s | If you're not a government person and you're playing spot the Fed, you have for sure come 1785.74s - 1787.42s | to the right room, and I'm really proud of you. 1789.18s - 1789.74s | Go ahead. 1792.79s - 1794.07s | Oh, that's a fair point. 1794.07s - 1797.03s | Everyone's a Fed, but go ahead, look to your left, look to your right. 1797.03s - 1797.67s


chunk_text:
there's more money and ongoing support fees in the second one, so most groups want to 1759.58s - 1763.66s | do that type of work and not the first group, so what you wind up with is people who are 1763.66s - 1769.82s | technically competent implementing and building things that might not be designed the best 1769.82s - 1771.58s | possible way, and that's not good either. 1772.54s - 1775.42s | I think it's just finding ways to engage and learn from industry and the tech community, 1775.42s - 1778.86s | so if you are in this room, you are doing that because you showed up, if you're a government 1778.86s - 1781.74s | person in this room anyway, you showed up to DEF CON, so good start. 1781.74s - 1785.74s | If you're not a government person and you're playing spot the Fed, you have for sure come 1785.74s - 1787.42s | to the right room, and I'm really proud of you. 1789.18s - 1789.74s | Go ahead. 1792.79s - 1794.07s | Oh, that's a fair point. 1794.07s - 1797.03s | Everyone's a Fed, but go ahead, look to your left, look to your right. 1797.03s - 1797.67s | They're both Feds. 1797.67s - 1800.71s | Talk to them, and then the last one is talk to Eddie. 1801.11s - 1805.27s | Don't have just an email black box where you put all contractors and never speak to them. 1805.91s - 1807.43s | Not just Eddie, but people like him. 1808.39s - 1810.15s | Go and talk and engage with those communities. 1810.15s - 1812.07s | I think there's a lot to be gained from that. 1812.07s - 1814.15s | Yeah, and we mentioned open source foundations. 1814.15s - 1819.03s | There's CNCF, Cloud Native Compute Foundation, Open Source Security Foundation. 1819.03s - 1822.39s | There's all sorts of working groups in there, like I mentioned before, so come find us, 1822.39s - 1824.23s | come join with us on a neutral ground. 1825.19s - 1827.11s | So that means everything is fucked. 1827.83s - 1828.47s | Or maybe not. 1830.34s - 1832.66s | Here's our last kind of what you can do to help. 1833.22s - 1836.26s | Have you ever considered a career in public service? 1836.90s - 1838.66s | And this isn't just a meme. 1838.66s - 1842.26s | There's ways that you can work


chunk_text:
| They're both Feds. 1797.67s - 1800.71s | Talk to them, and then the last one is talk to Eddie. 1801.11s - 1805.27s | Don't have just an email black box where you put all contractors and never speak to them. 1805.91s - 1807.43s | Not just Eddie, but people like him. 1808.39s - 1810.15s | Go and talk and engage with those communities. 1810.15s - 1812.07s | I think there's a lot to be gained from that. 1812.07s - 1814.15s | Yeah, and we mentioned open source foundations. 1814.15s - 1819.03s | There's CNCF, Cloud Native Compute Foundation, Open Source Security Foundation. 1819.03s - 1822.39s | There's all sorts of working groups in there, like I mentioned before, so come find us, 1822.39s - 1824.23s | come join with us on a neutral ground. 1825.19s - 1827.11s | So that means everything is fucked. 1827.83s - 1828.47s | Or maybe not. 1830.34s - 1832.66s | Here's our last kind of what you can do to help. 1833.22s - 1836.26s | Have you ever considered a career in public service? 1836.90s - 1838.66s | And this isn't just a meme. 1838.66s - 1842.26s | There's ways that you can work in this space without working directly for the government. 1843.22s - 1846.26s | Like myself, I joined a defense contractor to work in open source software, 1846.98s - 1853.38s | but there's other methods out there, like 18F from GSA is doing a great job to cut the hiring 1853.38s - 1856.66s | speed and kind of like up salary caps for folks. 1856.74s - 1861.22s | CISA is also doing a great job in this space to bring people in from external talent. 1861.22s - 1864.02s | So it's something that you should absolutely consider for your next job. 1865.38s - 1868.82s | The next one is actually like reading law and policy. 1868.82s - 1872.50s | I used to joke that the only thing I had going for me as an attorney was literacy, 1873.06s - 1875.46s | because there were tons of times where people are like, oh, you can't do this. 1875.46s - 1879.54s | And I would literally just open the policy that they were citing and read it, 1879.54s - 1882.50s | sometimes to them, and they would go, oh, I guess you could do that. 1882.50s - 1886.74s | It's like they could


chunk_text:
in this space without working directly for the government. 1843.22s - 1846.26s | Like myself, I joined a defense contractor to work in open source software, 1846.98s - 1853.38s | but there's other methods out there, like 18F from GSA is doing a great job to cut the hiring 1853.38s - 1856.66s | speed and kind of like up salary caps for folks. 1856.74s - 1861.22s | CISA is also doing a great job in this space to bring people in from external talent. 1861.22s - 1864.02s | So it's something that you should absolutely consider for your next job. 1865.38s - 1868.82s | The next one is actually like reading law and policy. 1868.82s - 1872.50s | I used to joke that the only thing I had going for me as an attorney was literacy, 1873.06s - 1875.46s | because there were tons of times where people are like, oh, you can't do this. 1875.46s - 1879.54s | And I would literally just open the policy that they were citing and read it, 1879.54s - 1882.50s | sometimes to them, and they would go, oh, I guess you could do that. 1882.50s - 1886.74s | It's like they could have done that step first without coming to me. 1886.74s - 1888.18s | Like that did not take a lawyer. 1888.18s - 1891.94s | So read it yourself, especially if you're going to bitch about it. 1891.94s - 1894.90s | You have a lot more credibility if you've actually read it. 1894.90s - 1897.78s | And then when you read it, if you identify things that like, look, no kidding, 1897.78s - 1902.50s | impede your ability to get something done, there are ways to change that 1903.30s - 1906.18s | from within the government and from outside of the government, 1906.18s - 1907.70s | specifically from within the government. 1907.70s - 1910.90s | I was able to change the law five different times before I left the federal government. 1911.46s - 1914.50s | Sometimes surprisingly quickly, other times after literally, 1914.50s - 1918.66s | literally a decade of bitching about the same thing consistently, 1918.66s - 1920.42s | like, but it can actually happen. 1920.42s - 1921.78s | There are ways to do it. 1921.78s - 1922.74s | It might not be worth it. 1922.74s - 1923.30s | It might suck. 1923.30s - 1925.54s | But there's also other


chunk_text:
have done that step first without coming to me. 1886.74s - 1888.18s | Like that did not take a lawyer. 1888.18s - 1891.94s | So read it yourself, especially if you're going to bitch about it. 1891.94s - 1894.90s | You have a lot more credibility if you've actually read it. 1894.90s - 1897.78s | And then when you read it, if you identify things that like, look, no kidding, 1897.78s - 1902.50s | impede your ability to get something done, there are ways to change that 1903.30s - 1906.18s | from within the government and from outside of the government, 1906.18s - 1907.70s | specifically from within the government. 1907.70s - 1910.90s | I was able to change the law five different times before I left the federal government. 1911.46s - 1914.50s | Sometimes surprisingly quickly, other times after literally, 1914.50s - 1918.66s | literally a decade of bitching about the same thing consistently, 1918.66s - 1920.42s | like, but it can actually happen. 1920.42s - 1921.78s | There are ways to do it. 1921.78s - 1922.74s | It might not be worth it. 1922.74s - 1923.30s | It might suck. 1923.30s - 1925.54s | But there's also other foundations that can help you do that too. 1925.54s - 1925.70s | Yeah. 1925.70s - 1928.74s | And if you don't want to take that initiative on your own, work with the EFF, 1928.74s - 1932.10s | work with the ACLU, there's groups out there that this is what they are designed to do, 1932.10s - 1934.74s | is do the right thing and fight for the right policy change. 1934.74s - 1939.86s | And a lot of this space is if we don't get well-designed technical policy in place, 1939.86s - 1944.74s | some random government contractor or some senator can propose something they have no idea about. 1944.74s - 1948.66s | So we need to get ahead of regulating and legislating a lot of this technology. 1949.78s - 1952.26s | And then the last one is, it kind of goes hand in hand with the others. 1952.26s - 1954.66s | Like, if something seems completely stupid, 1955.46s - 1960.42s | give it like a hot second before you just assume that everyone involved is completely dumb. 1960.42s - 1963.78s | And like, actually see if you can dig in and see what's driving it. 1963.78s - 1967.62s |


chunk_text:
foundations that can help you do that too. 1925.54s - 1925.70s | Yeah. 1925.70s - 1928.74s | And if you don't want to take that initiative on your own, work with the EFF, 1928.74s - 1932.10s | work with the ACLU, there's groups out there that this is what they are designed to do, 1932.10s - 1934.74s | is do the right thing and fight for the right policy change. 1934.74s - 1939.86s | And a lot of this space is if we don't get well-designed technical policy in place, 1939.86s - 1944.74s | some random government contractor or some senator can propose something they have no idea about. 1944.74s - 1948.66s | So we need to get ahead of regulating and legislating a lot of this technology. 1949.78s - 1952.26s | And then the last one is, it kind of goes hand in hand with the others. 1952.26s - 1954.66s | Like, if something seems completely stupid, 1955.46s - 1960.42s | give it like a hot second before you just assume that everyone involved is completely dumb. 1960.42s - 1963.78s | And like, actually see if you can dig in and see what's driving it. 1963.78s - 1967.62s | Because if you can identify the reason for a very stupid policy, 1967.62s - 1975.30s | then maybe you can more articulately offer something new that meets that underlying intent. 1975.30s - 1975.54s | Yeah. 1975.54s - 1978.50s | And you have to assume positive intent on a lot of this stuff, right? 1978.50s - 1982.74s | Like, no one is out there to kind of do bad things that make people's lives terrible in this space. 1984.02s - 1984.66s | I'm naive. 1985.70s - 1987.54s | But you have to think about that, right? 1987.54s - 1991.46s | Like, it might just be a bad implementation of a good idea originally. 1991.46s - 1995.06s | And last but not least, invite government folks to participate in the things that you're doing. 1995.06s - 1996.10s | Invite them to the table. 1996.10s - 1997.22s | They do exist. 1997.38s - 2001.22s | When I went to the Army base, I expected to see everyone walking around in fatigues 2001.22s - 2003.06s | and holding guns and rifles. 2003.06s - 2006.50s | And I think I saw one gun the entire time I was at the base at the front gate.


chunk_text:
Because if you can identify the reason for a very stupid policy, 1967.62s - 1975.30s | then maybe you can more articulately offer something new that meets that underlying intent. 1975.30s - 1975.54s | Yeah. 1975.54s - 1978.50s | And you have to assume positive intent on a lot of this stuff, right? 1978.50s - 1982.74s | Like, no one is out there to kind of do bad things that make people's lives terrible in this space. 1984.02s - 1984.66s | I'm naive. 1985.70s - 1987.54s | But you have to think about that, right? 1987.54s - 1991.46s | Like, it might just be a bad implementation of a good idea originally. 1991.46s - 1995.06s | And last but not least, invite government folks to participate in the things that you're doing. 1995.06s - 1996.10s | Invite them to the table. 1996.10s - 1997.22s | They do exist. 1997.38s - 2001.22s | When I went to the Army base, I expected to see everyone walking around in fatigues 2001.22s - 2003.06s | and holding guns and rifles. 2003.06s - 2006.50s | And I think I saw one gun the entire time I was at the base at the front gate. 2007.06s - 2010.18s | And everyone else was just kind of chilling as normal people in a town. 2010.18s - 2012.98s | So I had a lot of misconceptions that I worked through. 2012.98s - 2015.54s | And these folks are awesome and super smart. 2015.54s - 2018.34s | So with that, I think we're definitely out of time. 2018.34s - 2020.58s | We'll probably be out in the hallway to answer questions until... 2020.58s - 2021.86s | Oh, I'm getting a four-minute warning. 2022.50s - 2023.30s | Three-minute warning. 2023.30s - 2025.06s | We could probably take one question. 2026.02s - 2026.52s | Yes. 2045.64s - 2046.04s | Yeah. 2046.04s - 2048.04s | There's a lot of that going on. 2048.04s - 2053.80s | We have an open SSF policy day in DC on September 9th or whatever that Monday is. 2053.80s - 2055.88s | We're hoping to do a lot of good out of that. 2055.88s - 2060.76s | Ava, if you're in the room, Ava works for CISA doing some cool policy in that front. 2061.64s - 2062.36s | I don't see Ava. 2062.36s - 2065.40s | But yeah, there's a lot of efforts to try and


chunk_text:
2007.06s - 2010.18s | And everyone else was just kind of chilling as normal people in a town. 2010.18s - 2012.98s | So I had a lot of misconceptions that I worked through. 2012.98s - 2015.54s | And these folks are awesome and super smart. 2015.54s - 2018.34s | So with that, I think we're definitely out of time. 2018.34s - 2020.58s | We'll probably be out in the hallway to answer questions until... 2020.58s - 2021.86s | Oh, I'm getting a four-minute warning. 2022.50s - 2023.30s | Three-minute warning. 2023.30s - 2025.06s | We could probably take one question. 2026.02s - 2026.52s | Yes. 2045.64s - 2046.04s | Yeah. 2046.04s - 2048.04s | There's a lot of that going on. 2048.04s - 2053.80s | We have an open SSF policy day in DC on September 9th or whatever that Monday is. 2053.80s - 2055.88s | We're hoping to do a lot of good out of that. 2055.88s - 2060.76s | Ava, if you're in the room, Ava works for CISA doing some cool policy in that front. 2061.64s - 2062.36s | I don't see Ava. 2062.36s - 2065.40s | But yeah, there's a lot of efforts to try and make that better. 2066.12s - 2067.96s | And especially, like we said, the foundation work. 2067.96s - 2071.00s | Because if we can set up those agreements with the foundation and the DoD, 2071.08s - 2074.44s | then it no longer becomes a place of trying to sell people stuff. 2074.44s - 2078.84s | It becomes a place of how can we collaborate and actually make technical decisions that make sense. 2079.96s - 2081.32s | And with that, we really are out of time. 2081.32s - 2082.44s | Thank you guys so much. 2082.44s - 2083.96s | We will stand in the hallway. 2083.96s - 2086.76s | Thank you.


chunk_text:
make that better. 2066.12s - 2067.96s | And especially, like we said, the foundation work. 2067.96s - 2071.00s | Because if we can set up those agreements with the foundation and the DoD, 2071.08s - 2074.44s | then it no longer becomes a place of trying to sell people stuff. 2074.44s - 2078.84s | It becomes a place of how can we collaborate and actually make technical decisions that make sense. 2079.96s - 2081.32s | And with that, we really are out of time. 2081.32s - 2082.44s | Thank you guys so much. 2082.44s - 2083.96s | We will stand in the hallway. 2083.96s - 2086.76s | Thank you.


chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=BiBMsw0N_mQ", "title": "DEF CON 32 - Detecting persistent threats on Draytek devices - Octavio Gianatiempo, Gast\u00f3n Aznarez", "description": "Advanced attackers are increasingly choosing edge devices as targets, many of which are security appliances such as VPNs and Firewalls. They run closed-source firmware, and defenders and researchers must understand it to assess its security and integrity. We faced this firsthand when a client that used Draytek equipment was compromised. With at least 500k Draytek routers exposed to the Internet globally, no working tools exist to extract their firmware and assist researchers and defenders working with them.\n\nWe reverse-engineered Draytek's firmware format, developed tools to extract it, and discovered that its RTOS kernel can load code modules dynamically. These stored modules remain active even after firmware upgrades, inadvertently facilitating persistent threats. We crafted and uploaded malicious modules using our tools and newly found vulnerabilities to achieve persistence.\n\nEnd-users lack straightforward means to detect such compromises. In response to this threat, we developed our own module to assess the integrity of other modules loaded in memory, mitigating its impact. In our pursuit of a more secure internet, we are sharing our knowledge and opening our tools to the community, enabling observability, hardening, transparency, and vulnerability research on Draytek edge devices.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 2142, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 1.60s | This text was transcribed using whisper model: large-v2 Hi, everyone. 1.60s - 4.92s | There is more people than I expect at 10 AM. 4.92s - 9.08s | Thank you very much for being here. 9.08s - 12.84s | You are the ones who survived until the last day in DEFCON. 12.84s - 15.04s | Awesome. 15.04s - 18.12s | Welcome to Taking Off the Blindfold, 18.12s - 24.20s | Detecting Persistent Threats on Drytech Edge Devices. 24.20s - 26.72s | We are part of the Faraday Research Team. 26.72s - 29.60s | My partner here is Octavio Geno-Tiempo. 29.60s - 31.52s | I am Gaston Asnares. 31.52s - 36.20s | And we have some experience working with firmware. 36.20s - 40.16s | We have a foundation on computer science. 40.16s - 42.72s | And we like to play CDF. 42.72s - 49.33s | As a resume, we would like to break things and do research. 49.33s - 51.93s | So what we are going to see today. 51.93s - 54.77s | First, we are going to see


chunk_text:
and vulnerability research on Draytek edge devices.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 2142, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 1.60s | This text was transcribed using whisper model: large-v2 Hi, everyone. 1.60s - 4.92s | There is more people than I expect at 10 AM. 4.92s - 9.08s | Thank you very much for being here. 9.08s - 12.84s | You are the ones who survived until the last day in DEFCON. 12.84s - 15.04s | Awesome. 15.04s - 18.12s | Welcome to Taking Off the Blindfold, 18.12s - 24.20s | Detecting Persistent Threats on Drytech Edge Devices. 24.20s - 26.72s | We are part of the Faraday Research Team. 26.72s - 29.60s | My partner here is Octavio Geno-Tiempo. 29.60s - 31.52s | I am Gaston Asnares. 31.52s - 36.20s | And we have some experience working with firmware. 36.20s - 40.16s | We have a foundation on computer science. 40.16s - 42.72s | And we like to play CDF. 42.72s - 49.33s | As a resume, we would like to break things and do research. 49.33s - 51.93s | So what we are going to see today. 51.93s - 54.77s | First, we are going to see a motivation for this research 54.77s - 56.69s | and this talk. 56.69s - 58.49s | We are going to see some reverse engineering 58.49s - 60.85s | on drytech firmwares. 60.85s - 63.97s | We are going to talk about vulnerabilities and attack 63.97s - 65.93s | vectors. 65.93s - 68.45s | We are going to see mitigation strategies. 68.45s - 70.73s | And at least, we are going to see 70.73s - 74.82s | some conclusions and takeaways. 74.82s - 78.38s | OK, how this started. 78.38s - 82.70s | We had a call of a client saying that it was compromised. 82.70s - 85.66s | And when we came to the scene, we 85.66s - 90.34s | see a lot of outdated drytech equipment 90.34s - 92.42s | in the edge of the network. 92.42s - 99.26s | Most of them were SOHO models that runs our RTOS. 99.26s - 100.98s | And we started making some analysis. 100.98s - 104.26s | We found previous vulnerabilities 104.26s - 110.26s | with proven exploits on high-end models of the drytech routers 110.26s - 113.74s | that runs Linux. 113.74s - 116.78s | And we started making some questions


chunk_text:
a motivation for this research 54.77s - 56.69s | and this talk. 56.69s - 58.49s | We are going to see some reverse engineering 58.49s - 60.85s | on drytech firmwares. 60.85s - 63.97s | We are going to talk about vulnerabilities and attack 63.97s - 65.93s | vectors. 65.93s - 68.45s | We are going to see mitigation strategies. 68.45s - 70.73s | And at least, we are going to see 70.73s - 74.82s | some conclusions and takeaways. 74.82s - 78.38s | OK, how this started. 78.38s - 82.70s | We had a call of a client saying that it was compromised. 82.70s - 85.66s | And when we came to the scene, we 85.66s - 90.34s | see a lot of outdated drytech equipment 90.34s - 92.42s | in the edge of the network. 92.42s - 99.26s | Most of them were SOHO models that runs our RTOS. 99.26s - 100.98s | And we started making some analysis. 100.98s - 104.26s | We found previous vulnerabilities 104.26s - 110.26s | with proven exploits on high-end models of the drytech routers 110.26s - 113.74s | that runs Linux. 113.74s - 116.78s | And we started making some questions about it. 116.78s - 120.34s | Like, are these vulnerabilities exploitable 120.34s - 122.78s | on the SOHO routers? 122.78s - 126.02s | There is other vulnerabilities that are unknown. 126.02s - 129.58s | And the principal question was, were these routers 129.58s - 134.02s | the entry point of the attackers? 134.02s - 135.90s | OK, we know that there is an increasing 135.90s - 140.10s | threat of advanced attack targeting edge devices. 140.10s - 144.18s | We could name some examples, like the sewer rat that 144.18s - 148.14s | was a sophisticated malware targeting SOHO routers. 148.18s - 150.70s | We also could name the hiatus rat 150.70s - 153.46s | that was a malware targeting drytech routers, 153.46s - 157.98s | but the high-end models. 157.98s - 161.46s | And at least we could name Benin Sartine 161.46s - 168.46s | that was an NSA tool targeting Cisco routers that was leaked 168.46s - 173.36s | by the shadow brokers group. 173.36s - 178.68s | OK, so what is the attack surface of these routers? 178.68s - 186.96s | We found over 5,100 exposed devices, mostly in the UK. 186.96s - 189.64s | And we


chunk_text:
about it. 116.78s - 120.34s | Like, are these vulnerabilities exploitable 120.34s - 122.78s | on the SOHO routers? 122.78s - 126.02s | There is other vulnerabilities that are unknown. 126.02s - 129.58s | And the principal question was, were these routers 129.58s - 134.02s | the entry point of the attackers? 134.02s - 135.90s | OK, we know that there is an increasing 135.90s - 140.10s | threat of advanced attack targeting edge devices. 140.10s - 144.18s | We could name some examples, like the sewer rat that 144.18s - 148.14s | was a sophisticated malware targeting SOHO routers. 148.18s - 150.70s | We also could name the hiatus rat 150.70s - 153.46s | that was a malware targeting drytech routers, 153.46s - 157.98s | but the high-end models. 157.98s - 161.46s | And at least we could name Benin Sartine 161.46s - 168.46s | that was an NSA tool targeting Cisco routers that was leaked 168.46s - 173.36s | by the shadow brokers group. 173.36s - 178.68s | OK, so what is the attack surface of these routers? 178.68s - 186.96s | We found over 5,100 exposed devices, mostly in the UK. 186.96s - 189.64s | And we found a lot of vulnerabilities, 189.64s - 193.16s | from an information disclosure to an authenticated 193.16s - 194.52s | remote code execution. 194.52s - 198.57s | So there is a lot of stuff. 198.57s - 204.17s | OK, so we had a Vigor 2925 in our office, 204.17s - 209.05s | running MIPS 32 RTOS. 209.05s - 213.05s | And we wanted to start making some research on that, 213.05s - 215.09s | but we have a problem. 215.09s - 217.81s | The code is closed source. 217.81s - 222.41s | So first, we had to reverse engineer the firmware format. 222.41s - 225.37s | We had to extract the firmwares, understand 225.37s - 229.81s | previous vulnerabilities, and then look for new ones. 229.81s - 238.12s | And finally, determine if this was the attack vector or not. 238.12s - 240.32s | So the first thing we did was trying 240.32s - 242.48s | to extract this firmware using a minwalk, 242.68s - 243.32s | but it failed. 243.32s - 244.96s | It produced no results. 244.96s - 247.28s | So we tried to look for some specialized tools, 247.28s - 249.36s | and we found this one,


chunk_text:
found a lot of vulnerabilities, 189.64s - 193.16s | from an information disclosure to an authenticated 193.16s - 194.52s | remote code execution. 194.52s - 198.57s | So there is a lot of stuff. 198.57s - 204.17s | OK, so we had a Vigor 2925 in our office, 204.17s - 209.05s | running MIPS 32 RTOS. 209.05s - 213.05s | And we wanted to start making some research on that, 213.05s - 215.09s | but we have a problem. 215.09s - 217.81s | The code is closed source. 217.81s - 222.41s | So first, we had to reverse engineer the firmware format. 222.41s - 225.37s | We had to extract the firmwares, understand 225.37s - 229.81s | previous vulnerabilities, and then look for new ones. 229.81s - 238.12s | And finally, determine if this was the attack vector or not. 238.12s - 240.32s | So the first thing we did was trying 240.32s - 242.48s | to extract this firmware using a minwalk, 242.68s - 243.32s | but it failed. 243.32s - 244.96s | It produced no results. 244.96s - 247.28s | So we tried to look for some specialized tools, 247.28s - 249.36s | and we found this one, DRYTOOLS. 249.36s - 251.56s | So this didn't work with new firmwares. 251.56s - 255.04s | The last commit was 12 years ago, and that's why. 255.04s - 258.08s | So luckily for us, we found a talk 258.08s - 261.36s | by Philippe Loret at Hexagon 2022, 261.36s - 264.72s | where he described that by emulating DRYTEXT bootloader, 264.72s - 268.96s | we can extract, we can make it extract the RTOS kernel. 268.96s - 270.88s | The disadvantage of this approach 270.88s - 272.44s | is that this is a manual process, 272.44s - 274.80s | and you have to tweak the addresses for each firmware 274.80s - 276.60s | version. 276.60s - 278.24s | And the other problem that it has 278.24s - 280.56s | is that these firmware images have 280.56s - 284.36s | other executable components apart from the RTOS kernel 284.36s - 288.88s | and file systems that you cannot extract this way. 288.88s - 292.12s | So our approach was to reverse engineer the format 292.12s - 295.64s | and to try to write a tool that can extract all firmware 295.64s - 299.28s | versions from different models, extracting all file


chunk_text:
DRYTOOLS. 249.36s - 251.56s | So this didn't work with new firmwares. 251.56s - 255.04s | The last commit was 12 years ago, and that's why. 255.04s - 258.08s | So luckily for us, we found a talk 258.08s - 261.36s | by Philippe Loret at Hexagon 2022, 261.36s - 264.72s | where he described that by emulating DRYTEXT bootloader, 264.72s - 268.96s | we can extract, we can make it extract the RTOS kernel. 268.96s - 270.88s | The disadvantage of this approach 270.88s - 272.44s | is that this is a manual process, 272.44s - 274.80s | and you have to tweak the addresses for each firmware 274.80s - 276.60s | version. 276.60s - 278.24s | And the other problem that it has 278.24s - 280.56s | is that these firmware images have 280.56s - 284.36s | other executable components apart from the RTOS kernel 284.36s - 288.88s | and file systems that you cannot extract this way. 288.88s - 292.12s | So our approach was to reverse engineer the format 292.12s - 295.64s | and to try to write a tool that can extract all firmware 295.64s - 299.28s | versions from different models, extracting all file systems, 299.28s - 303.68s | and extracting all the executable components. 303.68s - 305.84s | We named our tool DRYTEXT Arsenal. 305.84s - 307.40s | It's a collection of tools. 307.40s - 309.48s | We wrote this in Python. 309.48s - 310.24s | It's open source. 310.24s - 313.88s | You can check it out at that repository. 313.88s - 317.20s | And the idea is to support for all these new firmware 317.20s - 319.96s | versions, and we will be showing, 319.96s - 322.28s | along these presentations, all the features 322.28s - 326.05s | that we implemented. 326.05s - 328.77s | So let's talk about the firmware format. 328.77s - 331.09s | It has two main sections. 331.09s - 332.69s | One is called the binary section, 332.69s - 334.77s | and the other one, the web section. 334.77s - 337.93s | Both start with a header that contains 337.93s - 340.85s | the size and a pointer to the following section. 340.85s - 343.21s | But in the first one, we also have information 343.21s - 347.45s | about the firmware version and some metadata. 347.45s - 352.49s | So the web section has a file


chunk_text:
systems, 299.28s - 303.68s | and extracting all the executable components. 303.68s - 305.84s | We named our tool DRYTEXT Arsenal. 305.84s - 307.40s | It's a collection of tools. 307.40s - 309.48s | We wrote this in Python. 309.48s - 310.24s | It's open source. 310.24s - 313.88s | You can check it out at that repository. 313.88s - 317.20s | And the idea is to support for all these new firmware 317.20s - 319.96s | versions, and we will be showing, 319.96s - 322.28s | along these presentations, all the features 322.28s - 326.05s | that we implemented. 326.05s - 328.77s | So let's talk about the firmware format. 328.77s - 331.09s | It has two main sections. 331.09s - 332.69s | One is called the binary section, 332.69s - 334.77s | and the other one, the web section. 334.77s - 337.93s | Both start with a header that contains 337.93s - 340.85s | the size and a pointer to the following section. 340.85s - 343.21s | But in the first one, we also have information 343.21s - 347.45s | about the firmware version and some metadata. 347.45s - 352.49s | So the web section has a file system that is compressed 352.49s - 354.61s | and contains all the static files 354.61s - 358.21s | that the router serves for the web UI. 358.21s - 360.73s | In the first section, the binary section, 360.73s - 364.61s | we have the boot loader that is raw MIPS 32 code. 364.61s - 367.53s | This is the part that Philippe emulated. 367.53s - 372.41s | And then this is followed by the compressed RTOS kernel. 372.41s - 375.37s | And after that, we have a very interesting part of firmware 375.37s - 377.09s | that is the DLM code. 377.09s - 381.37s | This is also a file system, but it's compressed and encrypted. 381.37s - 384.77s | And we will talk a lot about it. 384.77s - 389.09s | So to work with this, we made a specification 389.09s - 391.73s | of the structure of the firmware using KTI structs. 391.73s - 393.61s | And the advantage of using this is 393.61s - 396.69s | that you can manipulate the firmware images with any 396.69s - 398.33s | language that you want from this list. 398.33s - 400.73s | We used Python, but you can find


chunk_text:
system that is compressed 352.49s - 354.61s | and contains all the static files 354.61s - 358.21s | that the router serves for the web UI. 358.21s - 360.73s | In the first section, the binary section, 360.73s - 364.61s | we have the boot loader that is raw MIPS 32 code. 364.61s - 367.53s | This is the part that Philippe emulated. 367.53s - 372.41s | And then this is followed by the compressed RTOS kernel. 372.41s - 375.37s | And after that, we have a very interesting part of firmware 375.37s - 377.09s | that is the DLM code. 377.09s - 381.37s | This is also a file system, but it's compressed and encrypted. 381.37s - 384.77s | And we will talk a lot about it. 384.77s - 389.09s | So to work with this, we made a specification 389.09s - 391.73s | of the structure of the firmware using KTI structs. 391.73s - 393.61s | And the advantage of using this is 393.61s - 396.69s | that you can manipulate the firmware images with any 396.69s - 398.33s | language that you want from this list. 398.33s - 400.73s | We used Python, but you can find this in the repo 400.73s - 404.97s | and use the language that you prefer. 404.97s - 407.45s | Well, by knowing all this, we started 407.45s - 409.09s | to tackle the reverse engineering 409.09s - 411.13s | of the compression algorithm. 411.13s - 415.17s | So during boot, the RTOS is decompressed 415.17s - 418.49s | to the same memory space as the boot loader. 418.49s - 421.13s | We did this following the steps described 421.13s - 423.29s | by Philippe in his talk. 423.29s - 425.69s | But when we started reverse engineering this, 425.69s - 430.09s | we mapped all this in memory, the boot loader and then 430.09s - 431.89s | the RTOS after that. 431.89s - 435.29s | And that allowed us to discover that the decompression 435.29s - 438.69s | function is used during runtime to decompress other things, 438.69s - 440.93s | for example, the file systems. 440.93s - 443.37s | And we also discovered some interesting strings, 443.37s - 446.33s | like the one that you can see down to the right, that 446.33s - 449.09s | says lc4 decompression failed. 449.09s - 452.37s | But if it was lc4


chunk_text:
this in the repo 400.73s - 404.97s | and use the language that you prefer. 404.97s - 407.45s | Well, by knowing all this, we started 407.45s - 409.09s | to tackle the reverse engineering 409.09s - 411.13s | of the compression algorithm. 411.13s - 415.17s | So during boot, the RTOS is decompressed 415.17s - 418.49s | to the same memory space as the boot loader. 418.49s - 421.13s | We did this following the steps described 421.13s - 423.29s | by Philippe in his talk. 423.29s - 425.69s | But when we started reverse engineering this, 425.69s - 430.09s | we mapped all this in memory, the boot loader and then 430.09s - 431.89s | the RTOS after that. 431.89s - 435.29s | And that allowed us to discover that the decompression 435.29s - 438.69s | function is used during runtime to decompress other things, 438.69s - 440.93s | for example, the file systems. 440.93s - 443.37s | And we also discovered some interesting strings, 443.37s - 446.33s | like the one that you can see down to the right, that 446.33s - 449.09s | says lc4 decompression failed. 449.09s - 452.37s | But if it was lc4 decompression algorithm, 452.37s - 456.25s | why we couldn't start it with linwalk? 456.25s - 459.13s | And the reason for this is that while they 459.13s - 463.53s | use a standard lc4 compression block format, 463.53s - 468.61s | they have written a custom frame format. 468.61s - 472.65s | So it's just a custom magic that is 472.65s - 475.29s | followed by a succession of block size and block 475.29s - 477.49s | data multiple times. 477.49s - 482.57s | So we wrote a decompressor and a compressor for that format. 482.57s - 485.93s | And then we moved on to the most interesting part 485.93s - 488.69s | that is the DLM section. 488.69s - 492.53s | So it turns out that the DLM stands 492.53s - 494.17s | for Dynamic Kernel Modules. 494.17s - 496.77s | So this real-time operating system 496.77s - 501.01s | has the ability to load code modules during runtime 501.01s - 502.17s | dynamically. 502.17s - 506.57s | And this feature works independently 506.57s - 508.49s | from the firmware. 508.49s - 514.33s | So during boot, we can see that the modules that 514.33s - 517.37s | come


chunk_text:
decompression algorithm, 452.37s - 456.25s | why we couldn't start it with linwalk? 456.25s - 459.13s | And the reason for this is that while they 459.13s - 463.53s | use a standard lc4 compression block format, 463.53s - 468.61s | they have written a custom frame format. 468.61s - 472.65s | So it's just a custom magic that is 472.65s - 475.29s | followed by a succession of block size and block 475.29s - 477.49s | data multiple times. 477.49s - 482.57s | So we wrote a decompressor and a compressor for that format. 482.57s - 485.93s | And then we moved on to the most interesting part 485.93s - 488.69s | that is the DLM section. 488.69s - 492.53s | So it turns out that the DLM stands 492.53s - 494.17s | for Dynamic Kernel Modules. 494.17s - 496.77s | So this real-time operating system 496.77s - 501.01s | has the ability to load code modules during runtime 501.01s - 502.17s | dynamically. 502.17s - 506.57s | And this feature works independently 506.57s - 508.49s | from the firmware. 508.49s - 514.33s | So during boot, we can see that the modules that 514.33s - 517.37s | come with the firmware image are loaded. 517.37s - 518.89s | There are three modules. 518.89s - 523.13s | And they are called vr9-geo-ip, vr9-app-e. 523.13s - 527.49s | We will talk a lot about this one, and vr9-ssh. 527.49s - 530.13s | So the first one contains information 530.13s - 532.53s | about the geolocation of IPs. 532.53s - 536.61s | The second one is used to block traffic from the users that 536.61s - 538.93s | are in the LAN. 538.93s - 542.97s | And the last one contains the implementation of SSH server. 542.97s - 545.85s | And this is really interesting because this implementation 545.85s - 550.28s | is separate from the firmware. 550.28s - 554.40s | So these modules can be updated during runtime. 554.40s - 559.40s | And if you install a new version of App Enforcement Module 559.40s - 563.72s | or SSH, these are loaded first and prevent 563.72s - 567.36s | the modules that come with the firmware to be loaded. 567.36s - 571.04s | You can see down to the right that, for example, 571.04s - 574.36s | in this case, the vr9-app-e module was loaded first. 574.36s - 579.32s |


chunk_text:
with the firmware image are loaded. 517.37s - 518.89s | There are three modules. 518.89s - 523.13s | And they are called vr9-geo-ip, vr9-app-e. 523.13s - 527.49s | We will talk a lot about this one, and vr9-ssh. 527.49s - 530.13s | So the first one contains information 530.13s - 532.53s | about the geolocation of IPs. 532.53s - 536.61s | The second one is used to block traffic from the users that 536.61s - 538.93s | are in the LAN. 538.93s - 542.97s | And the last one contains the implementation of SSH server. 542.97s - 545.85s | And this is really interesting because this implementation 545.85s - 550.28s | is separate from the firmware. 550.28s - 554.40s | So these modules can be updated during runtime. 554.40s - 559.40s | And if you install a new version of App Enforcement Module 559.40s - 563.72s | or SSH, these are loaded first and prevent 563.72s - 567.36s | the modules that come with the firmware to be loaded. 567.36s - 571.04s | You can see down to the right that, for example, 571.04s - 574.36s | in this case, the vr9-app-e module was loaded first. 574.36s - 579.32s | And then when the router tries to load 579.32s - 584.28s | the version that comes in the firmware image, it will fail. 584.28s - 587.16s | So doing some more reverse engineering, 587.16s - 589.04s | we found out that the modules are compressed 589.04s - 594.76s | using the same algorithm that is used for the RTOS, 594.76s - 596.76s | but they are also encrypted. 596.76s - 599.20s | So we recognize that the algorithm used 599.20s - 602.44s | was a modified version of XDA. 602.44s - 605.40s | And this is a symmetric encryption algorithm. 605.40s - 608.44s | And what this means is that the keys for decrypting these DLMs 608.44s - 610.32s | are present in the firmware. 610.32s - 613.74s | So we looked for them, and we wrote a decryptor and encryptor 613.74s - 615.16s | for the modules. 615.16s - 618.48s | So if you take a DLM, you decrypt it using this, 618.48s - 622.52s | and you decompress it using the modified version of LC4, 622.52s - 625.16s | then what you get is a relocatable ELF. 625.16s - 627.48s | And this is very interesting, because all


chunk_text:
And then when the router tries to load 579.32s - 584.28s | the version that comes in the firmware image, it will fail. 584.28s - 587.16s | So doing some more reverse engineering, 587.16s - 589.04s | we found out that the modules are compressed 589.04s - 594.76s | using the same algorithm that is used for the RTOS, 594.76s - 596.76s | but they are also encrypted. 596.76s - 599.20s | So we recognize that the algorithm used 599.20s - 602.44s | was a modified version of XDA. 602.44s - 605.40s | And this is a symmetric encryption algorithm. 605.40s - 608.44s | And what this means is that the keys for decrypting these DLMs 608.44s - 610.32s | are present in the firmware. 610.32s - 613.74s | So we looked for them, and we wrote a decryptor and encryptor 613.74s - 615.16s | for the modules. 615.16s - 618.48s | So if you take a DLM, you decrypt it using this, 618.48s - 622.52s | and you decompress it using the modified version of LC4, 622.52s - 625.16s | then what you get is a relocatable ELF. 625.16s - 627.48s | And this is very interesting, because all these terminal 627.48s - 630.80s | functions that this ELF uses, they 630.80s - 632.60s | have to have their symbol names. 632.60s - 637.37s | So they are really easy to reverse engineer. 637.37s - 639.65s | And this also means that the firmware 639.65s - 641.37s | must have an ELF loader. 641.37s - 643.89s | So we looked for that code, and we found 643.89s - 646.65s | that there is also a symbol table. 646.65s - 649.65s | This symbol table was described before by Philippe Loret, 649.65s - 653.29s | but only as an aid for reverse engineering, 653.29s - 655.33s | because imagine that you have a table where 655.33s - 658.89s | you have function names followed by function pointers. 658.89s - 661.65s | But now we know that this table is actually 661.65s - 667.01s | used by the ELF loader to link these modules to the kernel. 667.01s - 671.57s | So the advantage that this gives to us 671.57s - 674.21s | is that if we manage to write our own modules, 674.21s - 676.37s | we can use all these functions without knowing 676.37s - 679.92s | their actual


chunk_text:
these terminal 627.48s - 630.80s | functions that this ELF uses, they 630.80s - 632.60s | have to have their symbol names. 632.60s - 637.37s | So they are really easy to reverse engineer. 637.37s - 639.65s | And this also means that the firmware 639.65s - 641.37s | must have an ELF loader. 641.37s - 643.89s | So we looked for that code, and we found 643.89s - 646.65s | that there is also a symbol table. 646.65s - 649.65s | This symbol table was described before by Philippe Loret, 649.65s - 653.29s | but only as an aid for reverse engineering, 653.29s - 655.33s | because imagine that you have a table where 655.33s - 658.89s | you have function names followed by function pointers. 658.89s - 661.65s | But now we know that this table is actually 661.65s - 667.01s | used by the ELF loader to link these modules to the kernel. 667.01s - 671.57s | So the advantage that this gives to us 671.57s - 674.21s | is that if we manage to write our own modules, 674.21s - 676.37s | we can use all these functions without knowing 676.37s - 679.92s | their actual addresses. 679.92s - 684.56s | So we implemented all these in our tool. 684.56s - 688.04s | We have two functions, one called 688.04s - 690.32s | parse firmware that will print a lot of information 690.32s - 692.52s | about the firmware image, and the other one 692.52s - 696.48s | called extract that will extract all these pieces of the image. 696.48s - 699.84s | It will decrypt and decompress all the modules. 699.84s - 704.44s | It will extract all the file systems for the web server, 704.44s - 708.00s | and it will generate a binary image that 708.00s - 711.04s | contains the bootloader followed by the RTOS kernel, 711.04s - 713.72s | but decompressed. 713.72s - 716.88s | So if you want to start reverse engineering this, 716.88s - 719.00s | you must know the loading address. 719.00s - 722.96s | And what we realize is that all Drytech bootloaders 722.96s - 725.72s | start by calling the decompression function. 725.72s - 727.52s | So this function is always located 727.52s - 729.48s | near the start of the image. 729.48s - 733.92s | And since all this has to be loaded aligned to


chunk_text:
addresses. 679.92s - 684.56s | So we implemented all these in our tool. 684.56s - 688.04s | We have two functions, one called 688.04s - 690.32s | parse firmware that will print a lot of information 690.32s - 692.52s | about the firmware image, and the other one 692.52s - 696.48s | called extract that will extract all these pieces of the image. 696.48s - 699.84s | It will decrypt and decompress all the modules. 699.84s - 704.44s | It will extract all the file systems for the web server, 704.44s - 708.00s | and it will generate a binary image that 708.00s - 711.04s | contains the bootloader followed by the RTOS kernel, 711.04s - 713.72s | but decompressed. 713.72s - 716.88s | So if you want to start reverse engineering this, 716.88s - 719.00s | you must know the loading address. 719.00s - 722.96s | And what we realize is that all Drytech bootloaders 722.96s - 725.72s | start by calling the decompression function. 725.72s - 727.52s | So this function is always located 727.52s - 729.48s | near the start of the image. 729.48s - 733.92s | And since all this has to be loaded aligned to the page, 733.92s - 737.44s | then we can search for the address of this compression 737.44s - 738.44s | function. 738.44s - 741.00s | And if we remove the last three numbers, 741.00s - 743.28s | what we get is the address, the loading 743.28s - 744.72s | address of the whole image. 744.72s - 749.48s | So with that, you can start your reverse engineering process. 749.48s - 752.36s | We also implemented this in a functionality 752.36s - 754.88s | called find loading address. 754.88s - 757.92s | And when we had all that, we moved on 757.92s - 760.68s | to try to find some vulnerabilities and attack 760.68s - 769.18s | vectors. 769.18s - 772.94s | OK, so let's talk something about vulnerabilities. 772.94s - 776.54s | We are going to show some of them. 776.54s - 779.06s | Starting with the basics, we found 779.06s - 782.02s | that they had an insecure password storage. 782.02s - 785.18s | That means they stored the passwords and the credentials 785.18s - 787.38s | in plain text in memory. 787.38s - 790.42s | So if someone had access to the memory, 790.42s - 795.26s | they


chunk_text:
the page, 733.92s - 737.44s | then we can search for the address of this compression 737.44s - 738.44s | function. 738.44s - 741.00s | And if we remove the last three numbers, 741.00s - 743.28s | what we get is the address, the loading 743.28s - 744.72s | address of the whole image. 744.72s - 749.48s | So with that, you can start your reverse engineering process. 749.48s - 752.36s | We also implemented this in a functionality 752.36s - 754.88s | called find loading address. 754.88s - 757.92s | And when we had all that, we moved on 757.92s - 760.68s | to try to find some vulnerabilities and attack 760.68s - 769.18s | vectors. 769.18s - 772.94s | OK, so let's talk something about vulnerabilities. 772.94s - 776.54s | We are going to show some of them. 776.54s - 779.06s | Starting with the basics, we found 779.06s - 782.02s | that they had an insecure password storage. 782.02s - 785.18s | That means they stored the passwords and the credentials 785.18s - 787.38s | in plain text in memory. 787.38s - 790.42s | So if someone had access to the memory, 790.42s - 795.26s | they could read the credentials or maybe change another exploit 795.26s - 798.62s | to leak them. 798.62s - 802.26s | We also found non-constant time comparison 802.26s - 805.66s | on the credential. 805.66s - 810.86s | So they use a function that takes different amounts of time 810.86s - 813.58s | to compare different passwords, depending 813.58s - 819.42s | on how much correct characters are in the string. 819.42s - 825.73s | And that leads to a side attack channel, a timing attack. 825.73s - 830.29s | OK, so DirectX implements a second step authentication 830.29s - 832.41s | that's really nice. 832.41s - 837.13s | But that was predictable, because the code generation 837.13s - 840.81s | was done using a self-random function. 840.81s - 848.57s | But with the seed, they use the time elapsed since boot. 848.57s - 853.09s | So if you know for how long the router is running, 853.09s - 858.81s | you could predict the second factor authentication codes. 858.81s - 864.09s | We also found an older reference in the HCP server 864.09s - 868.77s | that leads to the native service attack. 868.77s - 872.49s | But the interesting


chunk_text:
could read the credentials or maybe change another exploit 795.26s - 798.62s | to leak them. 798.62s - 802.26s | We also found non-constant time comparison 802.26s - 805.66s | on the credential. 805.66s - 810.86s | So they use a function that takes different amounts of time 810.86s - 813.58s | to compare different passwords, depending 813.58s - 819.42s | on how much correct characters are in the string. 819.42s - 825.73s | And that leads to a side attack channel, a timing attack. 825.73s - 830.29s | OK, so DirectX implements a second step authentication 830.29s - 832.41s | that's really nice. 832.41s - 837.13s | But that was predictable, because the code generation 837.13s - 840.81s | was done using a self-random function. 840.81s - 848.57s | But with the seed, they use the time elapsed since boot. 848.57s - 853.09s | So if you know for how long the router is running, 853.09s - 858.81s | you could predict the second factor authentication codes. 858.81s - 864.09s | We also found an older reference in the HCP server 864.09s - 868.77s | that leads to the native service attack. 868.77s - 872.49s | But the interesting in that is that it was exposed to one. 872.49s - 875.41s | So if the router was exposed to internet, 875.41s - 880.17s | you could trigger that non-reference. 880.17s - 883.37s | And if we combine both vulnerabilities, 883.37s - 887.37s | we could predict the second step authentication codes, 887.37s - 891.25s | because we could make the router crash. 891.25s - 895.29s | And then we know the time elapsed since boot, 895.29s - 898.89s | and we predict the codes. 898.89s - 903.05s | And that's what we are going to see in the next demo. 903.05s - 905.53s | We could see on the right the serial communication 905.53s - 906.21s | with the router. 909.89s - 912.09s | No, that's on the left. 912.09s - 915.09s | On the right top is the script that we run. 915.09s - 922.65s | It triggers the non-reference, and the router is rebooting. 922.65s - 925.77s | It will take two or three reboots, 925.77s - 929.81s | because it crashed hard. 929.81s - 934.53s | And on the right bottom, we have the web UI, 934.53s - 940.86s | where we are going to try


chunk_text:
in that is that it was exposed to one. 872.49s - 875.41s | So if the router was exposed to internet, 875.41s - 880.17s | you could trigger that non-reference. 880.17s - 883.37s | And if we combine both vulnerabilities, 883.37s - 887.37s | we could predict the second step authentication codes, 887.37s - 891.25s | because we could make the router crash. 891.25s - 895.29s | And then we know the time elapsed since boot, 895.29s - 898.89s | and we predict the codes. 898.89s - 903.05s | And that's what we are going to see in the next demo. 903.05s - 905.53s | We could see on the right the serial communication 905.53s - 906.21s | with the router. 909.89s - 912.09s | No, that's on the left. 912.09s - 915.09s | On the right top is the script that we run. 915.09s - 922.65s | It triggers the non-reference, and the router is rebooting. 922.65s - 925.77s | It will take two or three reboots, 925.77s - 929.81s | because it crashed hard. 929.81s - 934.53s | And on the right bottom, we have the web UI, 934.53s - 940.86s | where we are going to try to log in. 940.86s - 943.50s | So now it boots. 943.50s - 949.77s | We are going to wait until the router initializes itself. 950.61s - 960.61s | And now we are trying some different codes. 960.61s - 969.51s | And we were able to predict the code, and we are logged in. 969.51s - 973.59s | So Octavio talked something about the APP enforcement 973.59s - 975.47s | signature feature. 975.47s - 977.31s | It's really great. 977.31s - 980.11s | If you have an enterprise and you 980.11s - 985.23s | want to block to your employees on web or I don't know, 985.23s - 991.19s | you will update the signatures in order to block that traffic. 991.19s - 996.31s | And how this process is made, there are two ways. 996.31s - 997.75s | The first one is the manual. 997.75s - 1002.43s | We will see on the right bottom that says Import. 1002.43s - 1006.43s | If we click there, we could upload any file 1006.43s - 1008.99s | with some checks. 1008.99s - 1013.79s | And the second way is the automatic way, 1013.79s - 1018.47s | in which we have to


chunk_text:
to log in. 940.86s - 943.50s | So now it boots. 943.50s - 949.77s | We are going to wait until the router initializes itself. 950.61s - 960.61s | And now we are trying some different codes. 960.61s - 969.51s | And we were able to predict the code, and we are logged in. 969.51s - 973.59s | So Octavio talked something about the APP enforcement 973.59s - 975.47s | signature feature. 975.47s - 977.31s | It's really great. 977.31s - 980.11s | If you have an enterprise and you 980.11s - 985.23s | want to block to your employees on web or I don't know, 985.23s - 991.19s | you will update the signatures in order to block that traffic. 991.19s - 996.31s | And how this process is made, there are two ways. 996.31s - 997.75s | The first one is the manual. 997.75s - 1002.43s | We will see on the right bottom that says Import. 1002.43s - 1006.43s | If we click there, we could upload any file 1006.43s - 1008.99s | with some checks. 1008.99s - 1013.79s | And the second way is the automatic way, 1013.79s - 1018.47s | in which we have to select the update server 1018.47s - 1022.83s | or select the default one. 1022.83s - 1025.23s | And we have to schedule the updates. 1025.23s - 1030.03s | So for example, every day at some point of the day, 1030.03s - 1035.39s | it will check if there is some update for the signatures. 1035.39s - 1041.11s | And how they implement this, the signatures, 1041.11s - 1046.63s | you could think on JSON formats or YAML or something like that. 1046.63s - 1051.71s | But Drytech has the idea to implement this as a module. 1051.71s - 1054.39s | So to update the signatures, they 1054.39s - 1061.63s | upload an L file with code and everything. 1061.63s - 1064.47s | OK, let's talk about some vulnerabilities 1064.47s - 1066.47s | related to the modules. 1066.47s - 1073.35s | The first one is they not also update certificates with code, 1073.35s - 1076.63s | but they don't check for certificates 1076.63s - 1079.11s | or have SSL enforcement. 1079.11s - 1083.43s | So if someone manipulates the configuration of the router 1083.43s - 1086.47s | or the DNAs, maybe DNS hijacking, 1086.47s - 1091.59s | they could upload a


chunk_text:
select the update server 1018.47s - 1022.83s | or select the default one. 1022.83s - 1025.23s | And we have to schedule the updates. 1025.23s - 1030.03s | So for example, every day at some point of the day, 1030.03s - 1035.39s | it will check if there is some update for the signatures. 1035.39s - 1041.11s | And how they implement this, the signatures, 1041.11s - 1046.63s | you could think on JSON formats or YAML or something like that. 1046.63s - 1051.71s | But Drytech has the idea to implement this as a module. 1051.71s - 1054.39s | So to update the signatures, they 1054.39s - 1061.63s | upload an L file with code and everything. 1061.63s - 1064.47s | OK, let's talk about some vulnerabilities 1064.47s - 1066.47s | related to the modules. 1066.47s - 1073.35s | The first one is they not also update certificates with code, 1073.35s - 1076.63s | but they don't check for certificates 1076.63s - 1079.11s | or have SSL enforcement. 1079.11s - 1083.43s | So if someone manipulates the configuration of the router 1083.43s - 1086.47s | or the DNAs, maybe DNS hijacking, 1086.47s - 1091.59s | they could upload a module to all the Drytech routers 1091.59s - 1094.63s | that has this feature. 1094.63s - 1099.71s | And we have found two post-authentication endpoints. 1099.71s - 1103.63s | The first one is the one that I told in the last slide. 1103.63s - 1107.19s | And the second one is that we found an endpoint 1107.19s - 1112.39s | to upload configurations files that doesn't have some checks. 1112.39s - 1116.19s | So they allow us to upload an arbitrary DLM. 1116.19s - 1119.75s | Now it's not only the APPA module, 1119.75s - 1123.43s | but we could upload any modules that we want. 1123.43s - 1129.75s | Maybe we could, I don't know, upload another SSH module. 1129.79s - 1132.91s | So we have three ways for an attacker 1132.91s - 1134.63s | to install malicious modules. 1134.63s - 1136.83s | The first one is the post-authentication endpoints 1136.83s - 1141.19s | that we talked about in the last slide. 1141.19s - 1143.95s | We have the module upload process 1143.95s - 1147.59s | that is hijacking the configuration or the DNS. 1147.59s - 1149.71s | And the last one is the supply chain, 1149.71s - 1152.75s


chunk_text:
module to all the Drytech routers 1091.59s - 1094.63s | that has this feature. 1094.63s - 1099.71s | And we have found two post-authentication endpoints. 1099.71s - 1103.63s | The first one is the one that I told in the last slide. 1103.63s - 1107.19s | And the second one is that we found an endpoint 1107.19s - 1112.39s | to upload configurations files that doesn't have some checks. 1112.39s - 1116.19s | So they allow us to upload an arbitrary DLM. 1116.19s - 1119.75s | Now it's not only the APPA module, 1119.75s - 1123.43s | but we could upload any modules that we want. 1123.43s - 1129.75s | Maybe we could, I don't know, upload another SSH module. 1129.79s - 1132.91s | So we have three ways for an attacker 1132.91s - 1134.63s | to install malicious modules. 1134.63s - 1136.83s | The first one is the post-authentication endpoints 1136.83s - 1141.19s | that we talked about in the last slide. 1141.19s - 1143.95s | We have the module upload process 1143.95s - 1147.59s | that is hijacking the configuration or the DNS. 1147.59s - 1149.71s | And the last one is the supply chain, 1149.71s - 1152.75s | because the modules are very easy to craft. 1152.75s - 1158.07s | So they are easily appended to the firmware. 1158.07s - 1160.99s | And anyone with physical access or, I don't know, 1160.99s - 1163.63s | exploiting some vulnerability, they 1163.63s - 1171.94s | could upload the module before selling it. 1171.94s - 1173.94s | OK, so knowing all of this, we could 1173.94s - 1176.02s | start making some modules. 1176.02s - 1177.82s | We implemented two commands. 1177.82s - 1179.42s | The first one is the MIPS compile 1179.42s - 1186.94s | that is going to make a compilation of source code 1186.94s - 1193.86s | into a relocatable ELF of the architecture of MIPS32. 1193.86s - 1198.14s | And we also have another command that is up to DLM. 1198.14s - 1200.82s | As the name said, it will convert an ELF 1200.82s - 1203.46s | to a DLM, compressing and encrypting 1203.46s - 1205.30s | with the provided keys. 1205.30s - 1209.22s | So we could append some web headers 1209.22s - 1215.37s | so we could upload them through the web UI. 1215.37s - 1217.69s | So we are going to see a


chunk_text:
| because the modules are very easy to craft. 1152.75s - 1158.07s | So they are easily appended to the firmware. 1158.07s - 1160.99s | And anyone with physical access or, I don't know, 1160.99s - 1163.63s | exploiting some vulnerability, they 1163.63s - 1171.94s | could upload the module before selling it. 1171.94s - 1173.94s | OK, so knowing all of this, we could 1173.94s - 1176.02s | start making some modules. 1176.02s - 1177.82s | We implemented two commands. 1177.82s - 1179.42s | The first one is the MIPS compile 1179.42s - 1186.94s | that is going to make a compilation of source code 1186.94s - 1193.86s | into a relocatable ELF of the architecture of MIPS32. 1193.86s - 1198.14s | And we also have another command that is up to DLM. 1198.14s - 1200.82s | As the name said, it will convert an ELF 1200.82s - 1203.46s | to a DLM, compressing and encrypting 1203.46s - 1205.30s | with the provided keys. 1205.30s - 1209.22s | So we could append some web headers 1209.22s - 1215.37s | so we could upload them through the web UI. 1215.37s - 1217.69s | So we are going to see a demo. 1217.69s - 1225.73s | We could see on the left the version of the APPA module. 1225.73s - 1227.45s | On the right, in the middle, we are 1227.45s - 1232.01s | going to run a server, a custom server that 1232.01s - 1242.70s | is pointing the configuration of the auto-updates. 1242.70s - 1248.54s | So on the top right, we could see the serial interface 1248.54s - 1250.86s | with the router. 1250.86s - 1255.58s | OK, so once the server is up, we are going to send, 1255.58s - 1259.42s | we are going to trigger a new DLM. 1259.42s - 1260.14s | So it's triggered. 1260.14s - 1263.58s | We could see the router has fetched 1263.58s - 1265.62s | the new module in our server. 1265.62s - 1268.66s | We could see the Faraday logo because it's our module. 1268.66s - 1272.66s | And if we log in again, we could see 1272.66s - 1275.34s | that the version of the module has changed. 1275.34s - 1284.86s | And we have some logs that said that it don't allow this. 1284.86s - 1289.10s | OK, let's make something more funnier. 1289.10s - 1291.50s |


chunk_text:
demo. 1217.69s - 1225.73s | We could see on the left the version of the APPA module. 1225.73s - 1227.45s | On the right, in the middle, we are 1227.45s - 1232.01s | going to run a server, a custom server that 1232.01s - 1242.70s | is pointing the configuration of the auto-updates. 1242.70s - 1248.54s | So on the top right, we could see the serial interface 1248.54s - 1250.86s | with the router. 1250.86s - 1255.58s | OK, so once the server is up, we are going to send, 1255.58s - 1259.42s | we are going to trigger a new DLM. 1259.42s - 1260.14s | So it's triggered. 1260.14s - 1263.58s | We could see the router has fetched 1263.58s - 1265.62s | the new module in our server. 1265.62s - 1268.66s | We could see the Faraday logo because it's our module. 1268.66s - 1272.66s | And if we log in again, we could see 1272.66s - 1275.34s | that the version of the module has changed. 1275.34s - 1284.86s | And we have some logs that said that it don't allow this. 1284.86s - 1289.10s | OK, let's make something more funnier. 1289.10s - 1291.50s | We are going to make a SSH backdoor. 1291.50s - 1292.62s | Why not? 1292.62s - 1295.86s | Because we have the module. 1295.86s - 1301.46s | So the original behavior of the SSH module is the top one. 1301.46s - 1303.78s | When we have a new connection, it's 1303.78s - 1306.42s | going to call the authentication function. 1306.42s - 1308.42s | And the authentication function will 1308.42s - 1312.70s | say that it fails or it authenticates. 1312.70s - 1318.58s | But what we are going to do is when a new connection enters, 1318.58s - 1321.54s | when a new connection is open, that 1321.54s - 1323.62s | is going to call our hook. 1323.62s - 1326.62s | And it's going to check our credentials. 1326.62s - 1329.90s | So if they are right, it authenticates. 1329.90s - 1332.94s | If not, it's going to call the original function 1332.94s - 1334.78s | because we don't want to change all 1334.78s - 1339.40s | the behavior of the SSH module. 1339.40s - 1341.12s | How is the roadmap for this? 1341.12s - 1344.60s | First, we have to do a reverse engineer of the SSH module


chunk_text:
We are going to make a SSH backdoor. 1291.50s - 1292.62s | Why not? 1292.62s - 1295.86s | Because we have the module. 1295.86s - 1301.46s | So the original behavior of the SSH module is the top one. 1301.46s - 1303.78s | When we have a new connection, it's 1303.78s - 1306.42s | going to call the authentication function. 1306.42s - 1308.42s | And the authentication function will 1308.42s - 1312.70s | say that it fails or it authenticates. 1312.70s - 1318.58s | But what we are going to do is when a new connection enters, 1318.58s - 1321.54s | when a new connection is open, that 1321.54s - 1323.62s | is going to call our hook. 1323.62s - 1326.62s | And it's going to check our credentials. 1326.62s - 1329.90s | So if they are right, it authenticates. 1329.90s - 1332.94s | If not, it's going to call the original function 1332.94s - 1334.78s | because we don't want to change all 1334.78s - 1339.40s | the behavior of the SSH module. 1339.40s - 1341.12s | How is the roadmap for this? 1341.12s - 1344.60s | First, we have to do a reverse engineer of the SSH module 1344.60s - 1347.76s | to find that function that makes the authentication. 1347.76s - 1349.16s | We are going to skip that part. 1349.16s - 1351.72s | And we are going to say that the target function is the SSH 1351.72s - 1356.40s | account and LDAP auth function. 1356.40s - 1360.24s | After that, we have to write and compile the hook. 1360.24s - 1365.68s | We have to merge the hook with the original module. 1365.68s - 1371.52s | And we have to modify all the relocations in the entry table. 1371.52s - 1377.23s | And last, we have to upload the DLM. 1377.23s - 1379.07s | So on the right, we could see the source 1379.07s - 1384.55s | code of our hook function that takes the same arguments 1384.55s - 1390.83s | as the target function, an SSH instance, and a packet. 1390.83s - 1394.23s | And it will check for our credentials. 1394.23s - 1396.47s | If they are right, it will return one 1396.47s - 1398.43s | that says it authenticates. 1398.43s - 1401.55s | And if not, it will call the target function. 1404.59s - 1407.71s | OK, in order to compile this,


chunk_text:
1344.60s - 1347.76s | to find that function that makes the authentication. 1347.76s - 1349.16s | We are going to skip that part. 1349.16s - 1351.72s | And we are going to say that the target function is the SSH 1351.72s - 1356.40s | account and LDAP auth function. 1356.40s - 1360.24s | After that, we have to write and compile the hook. 1360.24s - 1365.68s | We have to merge the hook with the original module. 1365.68s - 1371.52s | And we have to modify all the relocations in the entry table. 1371.52s - 1377.23s | And last, we have to upload the DLM. 1377.23s - 1379.07s | So on the right, we could see the source 1379.07s - 1384.55s | code of our hook function that takes the same arguments 1384.55s - 1390.83s | as the target function, an SSH instance, and a packet. 1390.83s - 1394.23s | And it will check for our credentials. 1394.23s - 1396.47s | If they are right, it will return one 1396.47s - 1398.43s | that says it authenticates. 1398.43s - 1401.55s | And if not, it will call the target function. 1404.59s - 1407.71s | OK, in order to compile this, we use 1407.71s - 1412.58s | our command MIPS compile. 1412.58s - 1415.42s | And then we have to merge both binaries, 1415.42s - 1418.90s | both relocatable ELFs. 1418.90s - 1424.66s | So as I said, a relocatable ELF has all the symbols 1424.66s - 1426.70s | with no fixed address. 1426.70s - 1429.98s | So they had a relocation table that 1429.98s - 1438.42s | will set everything that uses a symbol has to be modified 1438.42s - 1442.62s | to point to that symbol after linking. 1442.62s - 1444.82s | So we have all the symbols there. 1444.82s - 1451.78s | And once we merge it, we have the symbols of both modules. 1451.78s - 1457.30s | We wrote a command to do this that 1457.30s - 1463.06s | is with the name of MIPS merge that takes two ELF files 1463.06s - 1468.70s | and as an output has one file. 1468.70s - 1472.86s | OK, now we have to modify the relocations table. 1472.86s - 1476.58s | So first, we find all the relocation 1476.58s - 1479.98s | that talks about the target function. 1479.98s - 1484.06s | We could see on the top right


chunk_text:
we use 1407.71s - 1412.58s | our command MIPS compile. 1412.58s - 1415.42s | And then we have to merge both binaries, 1415.42s - 1418.90s | both relocatable ELFs. 1418.90s - 1424.66s | So as I said, a relocatable ELF has all the symbols 1424.66s - 1426.70s | with no fixed address. 1426.70s - 1429.98s | So they had a relocation table that 1429.98s - 1438.42s | will set everything that uses a symbol has to be modified 1438.42s - 1442.62s | to point to that symbol after linking. 1442.62s - 1444.82s | So we have all the symbols there. 1444.82s - 1451.78s | And once we merge it, we have the symbols of both modules. 1451.78s - 1457.30s | We wrote a command to do this that 1457.30s - 1463.06s | is with the name of MIPS merge that takes two ELF files 1463.06s - 1468.70s | and as an output has one file. 1468.70s - 1472.86s | OK, now we have to modify the relocations table. 1472.86s - 1476.58s | So first, we find all the relocation 1476.58s - 1479.98s | that talks about the target function. 1479.98s - 1484.06s | We could see on the top right the two examples. 1484.06s - 1487.34s | We are going to use the first one. 1487.34s - 1494.26s | If we print the code that is on that address on the bottom 1494.26s - 1498.38s | left, we could see that the jump is a jump instruction. 1498.38s - 1503.34s | But the jump instruction only has his opcode 1503.34s - 1506.10s | but doesn't have an address where to jump. 1506.10s - 1508.18s | That is because the relocation entry 1508.18s - 1510.70s | will set which one is going to be 1510.70s - 1512.90s | that address on link in time. 1512.90s - 1516.02s | And we could see there that is the target function. 1516.02s - 1520.42s | So we modify that entry to give us 1520.42s - 1529.19s | a result the right that points to the hack function. 1529.19s - 1533.50s | So time to hack. 1533.50s - 1536.54s | OK, so on the top right, we are trying 1536.54s - 1539.18s | to log in with our credentials. 1539.18s - 1541.42s | But we could see that it fails. 1541.42s - 1549.51s | So on the bottom left, we upload our backdoor module. 1549.51s


chunk_text:
the two examples. 1484.06s - 1487.34s | We are going to use the first one. 1487.34s - 1494.26s | If we print the code that is on that address on the bottom 1494.26s - 1498.38s | left, we could see that the jump is a jump instruction. 1498.38s - 1503.34s | But the jump instruction only has his opcode 1503.34s - 1506.10s | but doesn't have an address where to jump. 1506.10s - 1508.18s | That is because the relocation entry 1508.18s - 1510.70s | will set which one is going to be 1510.70s - 1512.90s | that address on link in time. 1512.90s - 1516.02s | And we could see there that is the target function. 1516.02s - 1520.42s | So we modify that entry to give us 1520.42s - 1529.19s | a result the right that points to the hack function. 1529.19s - 1533.50s | So time to hack. 1533.50s - 1536.54s | OK, so on the top right, we are trying 1536.54s - 1539.18s | to log in with our credentials. 1539.18s - 1541.42s | But we could see that it fails. 1541.42s - 1549.51s | So on the bottom left, we upload our backdoor module. 1549.51s - 1552.51s | And once it's uploaded, we could see that the file 1552.51s - 1554.55s | was written successfully. 1554.55s - 1561.42s | But we have to reboot the router. 1561.42s - 1563.66s | And once it reboots, we are going 1563.66s - 1567.78s | to try again with our credentials. 1567.78s - 1577.65s | We could see that the module was loaded. 1577.65s - 1579.73s | OK, so we tried to log in. 1579.73s - 1582.29s | And we could see that we were in. 1582.29s - 1594.54s | And we could see some loss in the serial communication. 1594.54s - 1598.62s | So this attack that Gaston showed is really dangerous. 1598.62s - 1602.70s | Imagine that an attacker can craft a modified SSH module, 1602.70s - 1605.70s | upload it to your dry-tech router, 1605.70s - 1608.46s | and you have no way of detecting this. 1608.46s - 1611.58s | Moreover, if you update the firmware, 1611.58s - 1613.38s | the module will be still there. 1613.38s - 1615.82s | It will persist, and even will persist 1615.82s - 1617.70s | if you reset the configuration of the router, 1617.70s - 1619.58s | because it's stored


chunk_text:
- 1552.51s | And once it's uploaded, we could see that the file 1552.51s - 1554.55s | was written successfully. 1554.55s - 1561.42s | But we have to reboot the router. 1561.42s - 1563.66s | And once it reboots, we are going 1563.66s - 1567.78s | to try again with our credentials. 1567.78s - 1577.65s | We could see that the module was loaded. 1577.65s - 1579.73s | OK, so we tried to log in. 1579.73s - 1582.29s | And we could see that we were in. 1582.29s - 1594.54s | And we could see some loss in the serial communication. 1594.54s - 1598.62s | So this attack that Gaston showed is really dangerous. 1598.62s - 1602.70s | Imagine that an attacker can craft a modified SSH module, 1602.70s - 1605.70s | upload it to your dry-tech router, 1605.70s - 1608.46s | and you have no way of detecting this. 1608.46s - 1611.58s | Moreover, if you update the firmware, 1611.58s - 1613.38s | the module will be still there. 1613.38s - 1615.82s | It will persist, and even will persist 1615.82s - 1617.70s | if you reset the configuration of the router, 1617.70s - 1619.58s | because it's stored in a separate file 1619.58s - 1621.82s | system in the flash. 1621.82s - 1626.30s | So we started to think about some mitigation strategies. 1626.30s - 1629.62s | The first one, the obvious that we thought about, 1629.62s - 1631.14s | was just to remove the feature. 1631.14s - 1633.42s | Because who wants to have dynamic modules 1633.42s - 1634.86s | on their router? 1634.86s - 1638.58s | But there might be some users that use this feature. 1638.58s - 1642.70s | So well, instead of noping all the calls 1642.70s - 1648.30s | to module loading during boot and during the update process, 1648.30s - 1651.14s | we decided to think for a better solution. 1651.14s - 1657.10s | And we devised a mechanism for checking 1657.10s - 1661.22s | in-memory integrity of the other modules that are loaded. 1661.22s - 1664.26s | These mechanisms should fetch signatures from a server, 1664.26s - 1665.94s | because the modules will get updates. 1665.94s - 1668.38s | So you have to update the signatures. 1668.38s - 1672.90s | But this time, you have to check the certificates. 1672.90s - 1675.70s | So once you've got those signatures, 1675.70s -


chunk_text:
in a separate file 1619.58s - 1621.82s | system in the flash. 1621.82s - 1626.30s | So we started to think about some mitigation strategies. 1626.30s - 1629.62s | The first one, the obvious that we thought about, 1629.62s - 1631.14s | was just to remove the feature. 1631.14s - 1633.42s | Because who wants to have dynamic modules 1633.42s - 1634.86s | on their router? 1634.86s - 1638.58s | But there might be some users that use this feature. 1638.58s - 1642.70s | So well, instead of noping all the calls 1642.70s - 1648.30s | to module loading during boot and during the update process, 1648.30s - 1651.14s | we decided to think for a better solution. 1651.14s - 1657.10s | And we devised a mechanism for checking 1657.10s - 1661.22s | in-memory integrity of the other modules that are loaded. 1661.22s - 1664.26s | These mechanisms should fetch signatures from a server, 1664.26s - 1665.94s | because the modules will get updates. 1665.94s - 1668.38s | So you have to update the signatures. 1668.38s - 1672.90s | But this time, you have to check the certificates. 1672.90s - 1675.70s | So once you've got those signatures, 1675.70s - 1679.10s | you have to find the loaded modules in memory, 1679.10s - 1682.94s | calculate the actual signatures, and compare them. 1682.94s - 1684.98s | And if there is a mismatch, then you 1684.98s - 1687.46s | have to notify the user somehow. 1687.46s - 1689.46s | And all this functionality should 1689.46s - 1692.18s | be included with the firmware. 1692.18s - 1696.02s | So what we did, as a proof of concept, 1696.02s - 1698.22s | was to use the open source module 1699.06s - 1701.38s | to introduce this code to the router. 1701.38s - 1703.78s | Because it's much easier to write a module 1703.78s - 1705.50s | than to patch the firmware, because you 1705.50s - 1709.14s | have all these functions from the symbol table 1709.14s - 1710.94s | that you can use. 1710.94s - 1715.82s | So there are two things that we have to solve. 1715.82s - 1717.82s | The first one is how to calculate 1717.82s - 1721.26s | a signature for a valid DLM. 1721.26s - 1724.78s | And the second one is to find the actual DLMs in memory 1724.78s - 1726.86s | and do the comparison. 1726.86s


chunk_text:
1679.10s | you have to find the loaded modules in memory, 1679.10s - 1682.94s | calculate the actual signatures, and compare them. 1682.94s - 1684.98s | And if there is a mismatch, then you 1684.98s - 1687.46s | have to notify the user somehow. 1687.46s - 1689.46s | And all this functionality should 1689.46s - 1692.18s | be included with the firmware. 1692.18s - 1696.02s | So what we did, as a proof of concept, 1696.02s - 1698.22s | was to use the open source module 1699.06s - 1701.38s | to introduce this code to the router. 1701.38s - 1703.78s | Because it's much easier to write a module 1703.78s - 1705.50s | than to patch the firmware, because you 1705.50s - 1709.14s | have all these functions from the symbol table 1709.14s - 1710.94s | that you can use. 1710.94s - 1715.82s | So there are two things that we have to solve. 1715.82s - 1717.82s | The first one is how to calculate 1717.82s - 1721.26s | a signature for a valid DLM. 1721.26s - 1724.78s | And the second one is to find the actual DLMs in memory 1724.78s - 1726.86s | and do the comparison. 1726.86s - 1732.18s | So we focused on that, and we accepted some limitations. 1732.18s - 1735.86s | We will not check the signatures every time a module is 1735.86s - 1737.78s | loaded, as it should be. 1737.78s - 1739.22s | We will only check signatures when 1739.22s - 1741.38s | our defensive module is loaded. 1741.38s - 1746.62s | And we will not try to notify users for now. 1746.62s - 1749.06s | So the first problem that we have to solve 1749.06s - 1752.50s | is how to calculate a hash or a signature for a DLM. 1752.50s - 1756.74s | We'll hash each signature with md5, 1756.74s - 1761.06s | because md5 was available on this symbol table. 1761.06s - 1763.66s | And we will hash only the static sections, 1763.66s - 1767.10s | because the dynamic sections will change once loaded. 1767.10s - 1769.38s | If the static section contains data, 1769.38s - 1771.26s | we'll hash it right away. 1771.26s - 1774.62s | But if it contains code, then, as Gaston 1774.62s - 1778.18s | explained during the module installation when it's linked, 1778.18s - 1780.34s | all the jams and the function calls 1780.34s


chunk_text:
- 1732.18s | So we focused on that, and we accepted some limitations. 1732.18s - 1735.86s | We will not check the signatures every time a module is 1735.86s - 1737.78s | loaded, as it should be. 1737.78s - 1739.22s | We will only check signatures when 1739.22s - 1741.38s | our defensive module is loaded. 1741.38s - 1746.62s | And we will not try to notify users for now. 1746.62s - 1749.06s | So the first problem that we have to solve 1749.06s - 1752.50s | is how to calculate a hash or a signature for a DLM. 1752.50s - 1756.74s | We'll hash each signature with md5, 1756.74s - 1761.06s | because md5 was available on this symbol table. 1761.06s - 1763.66s | And we will hash only the static sections, 1763.66s - 1767.10s | because the dynamic sections will change once loaded. 1767.10s - 1769.38s | If the static section contains data, 1769.38s - 1771.26s | we'll hash it right away. 1771.26s - 1774.62s | But if it contains code, then, as Gaston 1774.62s - 1778.18s | explained during the module installation when it's linked, 1778.18s - 1780.34s | all the jams and the function calls 1780.34s - 1785.90s | will get the actual addresses of the functions written to them. 1785.90s - 1789.22s | So we will hash only the opcodes for these sections. 1789.22s - 1793.90s | And the opcodes in MIPS are the first six bits. 1793.90s - 1798.10s | So once we got that, we have to find all the modules that 1798.10s - 1799.54s | are loaded. 1799.54s - 1803.50s | When the ELF loader installs a module, 1803.50s - 1806.42s | it creates an object to represent the module. 1806.42s - 1808.66s | And the space for this module is allocated 1808.66s - 1811.70s | using the linear allocator of the router. 1811.70s - 1814.98s | So this allocator maintains a list 1814.98s - 1819.34s | of all the chunks that it has assigned. 1819.34s - 1823.06s | If we manage to find this object that represents the DLM, 1823.06s - 1825.78s | we'll get the DLM name, a pointer 1825.78s - 1828.50s | to the text section of that module, 1828.50s - 1830.32s | and a pointer to the main section 1830.32s - 1834.14s | that will get executed after the module is loaded. 1834.14s - 1839.06s | So the


chunk_text:
- 1785.90s | will get the actual addresses of the functions written to them. 1785.90s - 1789.22s | So we will hash only the opcodes for these sections. 1789.22s - 1793.90s | And the opcodes in MIPS are the first six bits. 1793.90s - 1798.10s | So once we got that, we have to find all the modules that 1798.10s - 1799.54s | are loaded. 1799.54s - 1803.50s | When the ELF loader installs a module, 1803.50s - 1806.42s | it creates an object to represent the module. 1806.42s - 1808.66s | And the space for this module is allocated 1808.66s - 1811.70s | using the linear allocator of the router. 1811.70s - 1814.98s | So this allocator maintains a list 1814.98s - 1819.34s | of all the chunks that it has assigned. 1819.34s - 1823.06s | If we manage to find this object that represents the DLM, 1823.06s - 1825.78s | we'll get the DLM name, a pointer 1825.78s - 1828.50s | to the text section of that module, 1828.50s - 1830.32s | and a pointer to the main section 1830.32s - 1834.14s | that will get executed after the module is loaded. 1834.14s - 1839.06s | So the loader copies all the sections 1839.06s - 1841.58s | in a predefined order, one after the other, 1841.58s - 1843.14s | after the text section. 1843.14s - 1845.62s | This order is usually defined by the compiler. 1845.62s - 1850.54s | And we will assume that this order doesn't change. 1850.54s - 1853.62s | So on the bottom, you can see a little bit 1853.62s - 1856.50s | of the code of this defensive module, the part where 1856.50s - 1861.18s | we traverse the list of chunks from this allocator. 1861.18s - 1866.70s | And we will check if each chunk corresponds to a module object. 1866.70s - 1868.58s | If the chunk corresponds to a module object, 1868.58s - 1870.30s | we will print some information. 1870.30s - 1873.54s | And if this module happens to be the SSH one, 1873.54s - 1878.06s | then we will calculate the hashes and compare them. 1878.06s - 1879.58s | So here's the demonstration. 1879.58s - 1883.18s | This router is the one that has been backdoored 1883.18s - 1886.02s | and has a modified version of the SSH. 1886.02s - 1890.38s | Then we upload our defensive module using the


chunk_text:
loader copies all the sections 1839.06s - 1841.58s | in a predefined order, one after the other, 1841.58s - 1843.14s | after the text section. 1843.14s - 1845.62s | This order is usually defined by the compiler. 1845.62s - 1850.54s | And we will assume that this order doesn't change. 1850.54s - 1853.62s | So on the bottom, you can see a little bit 1853.62s - 1856.50s | of the code of this defensive module, the part where 1856.50s - 1861.18s | we traverse the list of chunks from this allocator. 1861.18s - 1866.70s | And we will check if each chunk corresponds to a module object. 1866.70s - 1868.58s | If the chunk corresponds to a module object, 1868.58s - 1870.30s | we will print some information. 1870.30s - 1873.54s | And if this module happens to be the SSH one, 1873.54s - 1878.06s | then we will calculate the hashes and compare them. 1878.06s - 1879.58s | So here's the demonstration. 1879.58s - 1883.18s | This router is the one that has been backdoored 1883.18s - 1886.02s | and has a modified version of the SSH. 1886.02s - 1890.38s | Then we upload our defensive module using the UI on the left. 1890.38s - 1895.78s | And you can see right there that once it's uploaded, 1895.78s - 1897.46s | it will try to find all the modules, 1897.46s - 1898.86s | printing their information. 1898.86s - 1901.26s | And when it reaches the SSH module, 1901.26s - 1904.18s | it will start comparing the hashes for each section. 1904.18s - 1908.58s | And it will find that there is a mismatch between the hashes 1908.58s - 1912.38s | on the second section. 1912.38s - 1916.98s | So to wrap this up, some conclusions and takeaways. 1916.98s - 1919.50s | All these vulnerabilities were found 1919.50s - 1923.70s | by us analyzing only the bigger 2925 router. 1923.70s - 1925.50s | But they are present in several models 1925.50s - 1929.06s | because all the SOHO routers use this real-time operating 1929.06s - 1930.06s | system. 1930.06s - 1935.46s | And even some of the higher-end routers that run Linux 1935.46s - 1939.14s | emulate the same real-time operating system. 1939.14s - 1942.02s | So we found all these vulnerabilities and some more 1942.02s - 1943.38s | that we didn't show. 1943.38s - 1944.90s | And they are


chunk_text:
UI on the left. 1890.38s - 1895.78s | And you can see right there that once it's uploaded, 1895.78s - 1897.46s | it will try to find all the modules, 1897.46s - 1898.86s | printing their information. 1898.86s - 1901.26s | And when it reaches the SSH module, 1901.26s - 1904.18s | it will start comparing the hashes for each section. 1904.18s - 1908.58s | And it will find that there is a mismatch between the hashes 1908.58s - 1912.38s | on the second section. 1912.38s - 1916.98s | So to wrap this up, some conclusions and takeaways. 1916.98s - 1919.50s | All these vulnerabilities were found 1919.50s - 1923.70s | by us analyzing only the bigger 2925 router. 1923.70s - 1925.50s | But they are present in several models 1925.50s - 1929.06s | because all the SOHO routers use this real-time operating 1929.06s - 1930.06s | system. 1930.06s - 1935.46s | And even some of the higher-end routers that run Linux 1935.46s - 1939.14s | emulate the same real-time operating system. 1939.14s - 1942.02s | So we found all these vulnerabilities and some more 1942.02s - 1943.38s | that we didn't show. 1943.38s - 1944.90s | And they are all patched by now. 1944.90s - 1949.78s | But we don't have CBA IDs yet. 1949.78s - 1953.54s | And well, some conclusions on the offensive side. 1953.54s - 1956.58s | The code for this real-time operating system is huge. 1956.58s - 1958.42s | It's really complex. 1958.82s - 1961.62s | We only analyzed a little bit of it. 1961.62s - 1965.02s | Imagine that the functions that are present in the symbol table 1965.02s - 1968.66s | are maybe not up to 5% of all the functions 1968.66s - 1970.02s | in this firmware. 1970.02s - 1972.10s | So there are probably more vulnerabilities here 1972.10s - 1973.34s | to be found. 1973.34s - 1976.54s | And that is why we are open sourcing our tool. 1976.54s - 1978.70s | And we would love for you to try it. 1978.70s - 1981.06s | And we would love to know if it's useful 1981.06s - 1982.62s | and if you find something. 1982.62s - 1986.70s | So there's the repository if you want to check it out. 1986.70s - 1990.26s | And now some conclusions on the defensive side. 1990.26s - 1993.02s | We show that it's possible to create


chunk_text:
all patched by now. 1944.90s - 1949.78s | But we don't have CBA IDs yet. 1949.78s - 1953.54s | And well, some conclusions on the offensive side. 1953.54s - 1956.58s | The code for this real-time operating system is huge. 1956.58s - 1958.42s | It's really complex. 1958.82s - 1961.62s | We only analyzed a little bit of it. 1961.62s - 1965.02s | Imagine that the functions that are present in the symbol table 1965.02s - 1968.66s | are maybe not up to 5% of all the functions 1968.66s - 1970.02s | in this firmware. 1970.02s - 1972.10s | So there are probably more vulnerabilities here 1972.10s - 1973.34s | to be found. 1973.34s - 1976.54s | And that is why we are open sourcing our tool. 1976.54s - 1978.70s | And we would love for you to try it. 1978.70s - 1981.06s | And we would love to know if it's useful 1981.06s - 1982.62s | and if you find something. 1982.62s - 1986.70s | So there's the repository if you want to check it out. 1986.70s - 1990.26s | And now some conclusions on the defensive side. 1990.26s - 1993.02s | We show that it's possible to create a defensive module 1993.02s - 1995.66s | to check the integrity of other modules in memory. 1995.66s - 1998.10s | But in an ideal situation, this feature 1998.10s - 2000.22s | should be implemented by the vendor. 2000.22s - 2002.82s | It should fetch these updated signatures 2002.82s - 2006.54s | from a trusted server, verifying the certificates. 2006.54s - 2008.54s | And for all this to work, if this feature 2008.54s - 2011.14s | is included in the firmware, also 2011.14s - 2014.46s | we must have a secure blockchain to trust 2014.46s - 2017.76s | that this is working right. 2018.04s - 2019.80s | To close, some takeaways. 2019.80s - 2023.24s | We think that closed source firmware, as this example, 2023.24s - 2026.60s | is just another case of security by security. 2026.60s - 2028.76s | All these vulnerabilities that we have found 2028.76s - 2033.00s | are really simple, and they would 2033.00s - 2035.72s | have been reported sooner by the community 2035.72s - 2040.12s | if it were easier to reverse engineer these routers. 2040.12s - 2043.60s | So some things as plain text password storage, 2043.60s - 2047.16s | the use of insecure


chunk_text:
a defensive module 1993.02s - 1995.66s | to check the integrity of other modules in memory. 1995.66s - 1998.10s | But in an ideal situation, this feature 1998.10s - 2000.22s | should be implemented by the vendor. 2000.22s - 2002.82s | It should fetch these updated signatures 2002.82s - 2006.54s | from a trusted server, verifying the certificates. 2006.54s - 2008.54s | And for all this to work, if this feature 2008.54s - 2011.14s | is included in the firmware, also 2011.14s - 2014.46s | we must have a secure blockchain to trust 2014.46s - 2017.76s | that this is working right. 2018.04s - 2019.80s | To close, some takeaways. 2019.80s - 2023.24s | We think that closed source firmware, as this example, 2023.24s - 2026.60s | is just another case of security by security. 2026.60s - 2028.76s | All these vulnerabilities that we have found 2028.76s - 2033.00s | are really simple, and they would 2033.00s - 2035.72s | have been reported sooner by the community 2035.72s - 2040.12s | if it were easier to reverse engineer these routers. 2040.12s - 2043.60s | So some things as plain text password storage, 2043.60s - 2047.16s | the use of insecure functions to compare sensitive data, 2047.16s - 2050.04s | predictable random number generation, 2050.04s - 2053.28s | and even the lack of SSL certificate validation. 2053.28s - 2056.92s | So given all the increase in attacks 2056.92s - 2059.24s | that we are seeing to edge devices, 2059.24s - 2061.18s | it would be really important for vendors 2061.18s - 2064.40s | to include features that facilitate observability, 2064.40s - 2066.76s | vulnerability research by the community, 2066.76s - 2070.52s | and to implement threat detection solutions. 2070.52s - 2071.80s | So this is it. 2071.80s - 2074.36s | Thank you for your time, first for coming so early. 2074.36s - 2078.04s | And if you have any questions, please ask us. 2078.04s - 2080.00s | There's a mic here if you want to ask. 2083.08s - 2089.75s | First question gets a share. 2089.75s - 2112.44s | Well, thank you, guys. 2112.44s - 2114.76s | OK, the question was if there is memory protection 2114.76s - 2117.20s | between the modules and the RTOS, and there's not. 2117.20s - 2118.96s | All these runs on the same memory space. 2118.96s - 2122.28s | So from a module, you could


chunk_text:
functions to compare sensitive data, 2047.16s - 2050.04s | predictable random number generation, 2050.04s - 2053.28s | and even the lack of SSL certificate validation. 2053.28s - 2056.92s | So given all the increase in attacks 2056.92s - 2059.24s | that we are seeing to edge devices, 2059.24s - 2061.18s | it would be really important for vendors 2061.18s - 2064.40s | to include features that facilitate observability, 2064.40s - 2066.76s | vulnerability research by the community, 2066.76s - 2070.52s | and to implement threat detection solutions. 2070.52s - 2071.80s | So this is it. 2071.80s - 2074.36s | Thank you for your time, first for coming so early. 2074.36s - 2078.04s | And if you have any questions, please ask us. 2078.04s - 2080.00s | There's a mic here if you want to ask. 2083.08s - 2089.75s | First question gets a share. 2089.75s - 2112.44s | Well, thank you, guys. 2112.44s - 2114.76s | OK, the question was if there is memory protection 2114.76s - 2117.20s | between the modules and the RTOS, and there's not. 2117.20s - 2118.96s | All these runs on the same memory space. 2118.96s - 2122.28s | So from a module, you could modify the RTOS code 2122.28s - 2135.26s | and vice versa. 2135.26s - 2137.78s | Well, thanks again for coming.


chunk_text:
modify the RTOS code 2122.28s - 2135.26s | and vice versa. 2135.26s - 2137.78s | Well, thanks again for coming.


chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=AS_bSDxTU4w", "title": "DEF CON 32 - Sneaky Extensions The MV3 Escape Artists - Vivek Ramachandran, Shourya Pratap Singh", "description": "Ever since the pandemic and the rising popularity of work-from-home and hybrid models, there has been an increase in the usage of browsers, particularly video conferencing and collaboration applications. While some extensions enhance the user experience, some can gravely affect users' privacy and security.\n\nOver the past few years, extensions have gained recognition for nefarious activities, from simple color picker extensions to productivity-first AI extensions. And now more than ever, attackers are leveraging malicious extensions to steal user data, promote ads, affiliate marketing, and more. Realizing the abuse, Google pivoted from the MV2 model to the latest MV3, providing better security and locking down the extension from running rampant. While some security measures have been introduced in MV3, it is far from safe. In this talk, we will be demonstrating a suite of attacks, while requiring the least amount of permissions, which 95% of extensions on the Chrome store have. We will showcase stealth stealing of webcam feed, audio streams, clipboard data, and stealing credentials from other extensions like password managers.\n\nMV3 also introduced security measures to block the usage of functions like eval and new Function that allowed arbitrary code execution. We\u2019ll showcase how an extension can still do arbitrary code execution effectively bypassing the MV3 restrictions.\n\nIn this talk, we will also propose changes to the extension security model to prevent the lurking loopholes. We will also be demonstrating how malicious extensions can interfere with other extensions and steal sensitive information such as Credit card, passwords, OTP, etc, from other extensions.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1231, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 5.52s | This text was transcribed using whisper model: large-v2 Morning everyone. Thank you so much for coming. I mean I know this is the very 5.52s - 9.64s | first talk happening literally in every single screen out there. So really 9.64s - 15.28s | appreciate you making it out to ours. I'm Vivek Ramachandran. I have Shaurya with me. 15.28s - 19.92s | And we're going to talk about something very interesting. Sneaky extensions. The 19.92s - 25.44s | MV3 escape artists. Little bit about me. I've been in cyber security 20 years. 25.44s - 32.00s | Very first Defcon 2007 when I spoke. Found multiple vulnerabilities. Started 32.00s - 36.48s | multiple cyber


chunk_text:
functions like eval and new Function that allowed arbitrary code execution. We\u2019ll showcase how an extension can still do arbitrary code execution effectively bypassing the MV3 restrictions.\n\nIn this talk, we will also propose changes to the extension security model to prevent the lurking loopholes. We will also be demonstrating how malicious extensions can interfere with other extensions and steal sensitive information such as Credit card, passwords, OTP, etc, from other extensions.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1231, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 5.52s | This text was transcribed using whisper model: large-v2 Morning everyone. Thank you so much for coming. I mean I know this is the very 5.52s - 9.64s | first talk happening literally in every single screen out there. So really 9.64s - 15.28s | appreciate you making it out to ours. I'm Vivek Ramachandran. I have Shaurya with me. 15.28s - 19.92s | And we're going to talk about something very interesting. Sneaky extensions. The 19.92s - 25.44s | MV3 escape artists. Little bit about me. I've been in cyber security 20 years. 25.44s - 32.00s | Very first Defcon 2007 when I spoke. Found multiple vulnerabilities. Started 32.00s - 36.48s | multiple cyber security companies. Now we run Squarex, a browser native security 36.48s - 43.14s | company. Hello everyone. I'm Shaurya. I work as a principal software engineer at 43.14s - 47.80s | Squarex. My work mainly involves researching extensions, malicious 47.80s - 54.81s | extensions and working on methods to actually tackle them. And we have Sasha 54.93s - 61.05s | from the speaker ops team. Okay so browser extensions. I think almost 61.05s - 65.77s | everyone uses it. Anyone here doesn't use browser extensions? I'm guessing maybe in 65.77s - 71.05s | this audience a few people are skeptical. But overall I guess everybody uses 71.05s - 76.01s | browser extensions. Good convenience feature. And that is really where I think 76.01s - 82.29s | our talk is about how easy it is for a browser extension to cause backdoors, a 82.29s - 87.45s | lot of attacks within your organization. And the grand finale is we'll actually 87.45s - 93.09s | show you a browser extension which we've written. Which can steal live calls on 93.09s - 97.69s | you know Google Meet, Zoom and all of that while the call is happening. Without 97.69s - 104.01s |


chunk_text:
security companies. Now we run Squarex, a browser native security 36.48s - 43.14s | company. Hello everyone. I'm Shaurya. I work as a principal software engineer at 43.14s - 47.80s | Squarex. My work mainly involves researching extensions, malicious 47.80s - 54.81s | extensions and working on methods to actually tackle them. And we have Sasha 54.93s - 61.05s | from the speaker ops team. Okay so browser extensions. I think almost 61.05s - 65.77s | everyone uses it. Anyone here doesn't use browser extensions? I'm guessing maybe in 65.77s - 71.05s | this audience a few people are skeptical. But overall I guess everybody uses 71.05s - 76.01s | browser extensions. Good convenience feature. And that is really where I think 76.01s - 82.29s | our talk is about how easy it is for a browser extension to cause backdoors, a 82.29s - 87.45s | lot of attacks within your organization. And the grand finale is we'll actually 87.45s - 93.09s | show you a browser extension which we've written. Which can steal live calls on 93.09s - 97.69s | you know Google Meet, Zoom and all of that while the call is happening. Without 97.69s - 104.01s | having any indication that that MITM interception is going on. So as I said 104.01s - 107.53s | everybody uses extensions. Grammarly, 1password, you know some of the more 107.61s - 113.33s | common ones. And now because of AI, every second day you keep hearing about 113.33s - 118.57s | somebody offering a free extension which gives you access to chat GPT premium. 118.57s - 122.85s | Right? And then people just go about installing it thinking they're getting 122.85s - 129.54s | free access. But many of those extensions are backdoored. Now this 129.54s - 134.26s | problem is starting to compound. So you know some statistics from Forbes where 134.26s - 140.10s | over 280 million installs of extensions containing malware in just three years. 140.10s - 147.39s | Just 32 of them managed to go ahead and have a download count of over 75 million. 147.39s - 152.67s | I think a lot of folks think that the Chrome store does a lot of analysis but 152.67s - 156.63s | it doesn't. It's very high level, very basic sanity checks and nothing more 156.63s - 161.59s | than that. And that's why this problem is starting


chunk_text:
having any indication that that MITM interception is going on. So as I said 104.01s - 107.53s | everybody uses extensions. Grammarly, 1password, you know some of the more 107.61s - 113.33s | common ones. And now because of AI, every second day you keep hearing about 113.33s - 118.57s | somebody offering a free extension which gives you access to chat GPT premium. 118.57s - 122.85s | Right? And then people just go about installing it thinking they're getting 122.85s - 129.54s | free access. But many of those extensions are backdoored. Now this 129.54s - 134.26s | problem is starting to compound. So you know some statistics from Forbes where 134.26s - 140.10s | over 280 million installs of extensions containing malware in just three years. 140.10s - 147.39s | Just 32 of them managed to go ahead and have a download count of over 75 million. 147.39s - 152.67s | I think a lot of folks think that the Chrome store does a lot of analysis but 152.67s - 156.63s | it doesn't. It's very high level, very basic sanity checks and nothing more 156.63s - 161.59s | than that. And that's why this problem is starting to compound more and more year 161.59s - 170.91s | over year. So give it to Shaurya to continue. So how many of you have ever 170.91s - 176.83s | written an extension before? Can you raise your hand? Okay. So I'll just tell 176.83s - 181.51s | you all a little bit about how an extension code structure looks like. This 181.51s - 184.63s | will actually help us understand how the whole permission model is 184.63s - 190.43s | and how exactly people can leverage that against. So typically any extension has 190.51s - 194.23s | these four things. So they have a manifest file, then they have a service 194.23s - 197.83s | worker which runs on the background and then they have a content script. So 197.83s - 203.35s | content script is basically injected on whatever web page you're opening. Now 203.35s - 207.51s | what all pages it is allowed to be present is specified in the manifest 207.51s - 213.55s | file. Lastly some extensions also have an HTML page. So let's say you go ahead and 213.55s - 216.99s | click on an icon, you can see like a small pop-up will


chunk_text:
to compound more and more year 161.59s - 170.91s | over year. So give it to Shaurya to continue. So how many of you have ever 170.91s - 176.83s | written an extension before? Can you raise your hand? Okay. So I'll just tell 176.83s - 181.51s | you all a little bit about how an extension code structure looks like. This 181.51s - 184.63s | will actually help us understand how the whole permission model is 184.63s - 190.43s | and how exactly people can leverage that against. So typically any extension has 190.51s - 194.23s | these four things. So they have a manifest file, then they have a service 194.23s - 197.83s | worker which runs on the background and then they have a content script. So 197.83s - 203.35s | content script is basically injected on whatever web page you're opening. Now 203.35s - 207.51s | what all pages it is allowed to be present is specified in the manifest 207.51s - 213.55s | file. Lastly some extensions also have an HTML page. So let's say you go ahead and 213.55s - 216.99s | click on an icon, you can see like a small pop-up will appear. So there's 216.99s - 221.35s | actually an HTML page. Similarly some extensions can also have like a separate 221.35s - 226.59s | page which you can access through your address bar by chrome hyphen extension 226.59s - 232.23s | colon slash slash and specifying the extension ID. So this is what a typical 232.23s - 237.15s | extension structure look like. Now to understand how the whole permission 237.15s - 243.03s | model works, it's important to note two things. The first one is that you know in 243.11s - 247.51s | the extension the very best thing that has been done is that you have to 247.51s - 252.55s | specify what all permissions are required you know upfront. So your 252.55s - 256.99s | manifest or JSON file itself specifies that what kind of permissions your 256.99s - 262.95s | extension is going to ask for. And this actually helps a lot as well. Now after 262.95s - 268.11s | this we also have this a small you know a snippet. This snippet is actually a 268.11s - 271.55s | manifest file. So this is how it typically looks like. If you see on the 271.55s


chunk_text:
appear. So there's 216.99s - 221.35s | actually an HTML page. Similarly some extensions can also have like a separate 221.35s - 226.59s | page which you can access through your address bar by chrome hyphen extension 226.59s - 232.23s | colon slash slash and specifying the extension ID. So this is what a typical 232.23s - 237.15s | extension structure look like. Now to understand how the whole permission 237.15s - 243.03s | model works, it's important to note two things. The first one is that you know in 243.11s - 247.51s | the extension the very best thing that has been done is that you have to 247.51s - 252.55s | specify what all permissions are required you know upfront. So your 252.55s - 256.99s | manifest or JSON file itself specifies that what kind of permissions your 256.99s - 262.95s | extension is going to ask for. And this actually helps a lot as well. Now after 262.95s - 268.11s | this we also have this a small you know a snippet. This snippet is actually a 268.11s - 271.55s | manifest file. So this is how it typically looks like. If you see on the 271.55s - 276.11s | top this is where you specify the manifest version. So the current latest 276.11s - 282.07s | version is version 3. Now you know this has been actually a very I would say 282.07s - 286.67s | important transition as extensions are moving from version 2 to version 3. 286.67s - 291.35s | Because as you can see here you know earlier in the permissions we had to 291.35s - 295.67s | specify a URL. Like you know on this particular URL you have access to. But 295.67s - 299.75s | now it's much more clearer. So it's been broken up into further keys to bring 299.75s - 304.67s | much more clarity on how the permission is being handled. Now if we look at the 304.67s - 308.07s | kind of permissions most of the extensions are asking for here are some 308.07s - 313.35s | stats. So you can see that the third most is actually all URLs which is quite I 313.35s - 318.15s | would say dangerous because most of your extension actually have access to 318.15s - 324.13s | modify any page if they want. Most people don't even read what the


chunk_text:
- 276.11s | top this is where you specify the manifest version. So the current latest 276.11s - 282.07s | version is version 3. Now you know this has been actually a very I would say 282.07s - 286.67s | important transition as extensions are moving from version 2 to version 3. 286.67s - 291.35s | Because as you can see here you know earlier in the permissions we had to 291.35s - 295.67s | specify a URL. Like you know on this particular URL you have access to. But 295.67s - 299.75s | now it's much more clearer. So it's been broken up into further keys to bring 299.75s - 304.67s | much more clarity on how the permission is being handled. Now if we look at the 304.67s - 308.07s | kind of permissions most of the extensions are asking for here are some 308.07s - 313.35s | stats. So you can see that the third most is actually all URLs which is quite I 313.35s - 318.15s | would say dangerous because most of your extension actually have access to 318.15s - 324.13s | modify any page if they want. Most people don't even read what the permission you 324.13s - 329.03s | know the prompt is saying to you. Now the second important thing to know is how 329.03s - 334.43s | the whole process architecture inside the Chromium works like. So in Chromium 334.43s - 340.23s | we have a multi process architecture. So we have separate processes for doing 340.23s - 345.43s | separate things. For example you know if you have like a particular tab so 345.43s - 348.59s | whatever you're seeing on the page is being done through a renderer process. 348.59s - 352.39s | Similarly if there's a network call it will actually happen through an entirely 352.39s - 357.79s | different process. So by having all these processes separated you know it really 357.83s - 363.75s | helps out in a security kind of point. The reason being that all these are 363.75s - 368.87s | basically isolated from the operating system point of view 368.87s - 374.55s | because these cannot really you know interact at much other than the IPCs. Now 374.55s - 380.15s | there is one more thing which is quite interesting. So every tab that you see is 380.15s - 385.67s | actually having


chunk_text:
permission you 324.13s - 329.03s | know the prompt is saying to you. Now the second important thing to know is how 329.03s - 334.43s | the whole process architecture inside the Chromium works like. So in Chromium 334.43s - 340.23s | we have a multi process architecture. So we have separate processes for doing 340.23s - 345.43s | separate things. For example you know if you have like a particular tab so 345.43s - 348.59s | whatever you're seeing on the page is being done through a renderer process. 348.59s - 352.39s | Similarly if there's a network call it will actually happen through an entirely 352.39s - 357.79s | different process. So by having all these processes separated you know it really 357.83s - 363.75s | helps out in a security kind of point. The reason being that all these are 363.75s - 368.87s | basically isolated from the operating system point of view 368.87s - 374.55s | because these cannot really you know interact at much other than the IPCs. Now 374.55s - 380.15s | there is one more thing which is quite interesting. So every tab that you see is 380.15s - 385.67s | actually having a different renderer process which actually further isolates 385.71s - 390.11s | them from interacting with each other. Now you must be wondering you know what 390.11s - 395.21s | happens if there's an iframe on the page right. So if a page has an iframe it is 395.21s - 399.63s | further put in a different process. It's actually I would say a very big 399.63s - 403.83s | engineering milestone because if you simply go to a web page and type you 403.83s - 408.07s | know a command F to find a text it's actually searching that text across 408.07s - 413.31s | multiple processes. So it's quite an engineering feat I would say. There are 413.31s - 417.99s | different terms for this called a site isolation or site fission. Now we'll come 417.99s - 421.71s | back to how the whole you know this architecture fits in. So this diagram 421.71s - 425.27s | basically shows it pretty well. So all the components which I showed you 425.27s - 428.55s | earlier like extension pages, your content script, this is how the whole 428.55s - 434.51s | structure looks like. Now content script are I


chunk_text:
a different renderer process which actually further isolates 385.71s - 390.11s | them from interacting with each other. Now you must be wondering you know what 390.11s - 395.21s | happens if there's an iframe on the page right. So if a page has an iframe it is 395.21s - 399.63s | further put in a different process. It's actually I would say a very big 399.63s - 403.83s | engineering milestone because if you simply go to a web page and type you 403.83s - 408.07s | know a command F to find a text it's actually searching that text across 408.07s - 413.31s | multiple processes. So it's quite an engineering feat I would say. There are 413.31s - 417.99s | different terms for this called a site isolation or site fission. Now we'll come 417.99s - 421.71s | back to how the whole you know this architecture fits in. So this diagram 421.71s - 425.27s | basically shows it pretty well. So all the components which I showed you 425.27s - 428.55s | earlier like extension pages, your content script, this is how the whole 428.55s - 434.51s | structure looks like. Now content script are I would say the very first thing 434.51s - 439.43s | that we all should you know be careful about because content scripts are 439.43s - 444.91s | injected on the web pages. Now to see it in action I'll actually show you a live 444.91s - 449.35s | code right now to see how exactly content script can modify a given page. 449.35s - 456.63s | So it's actually pretty straightforward. This is a typical extension. You can see 456.63s - 461.59s | like there's a manifest file. I'm just naming it test extension for now and in 461.59s - 466.51s | this I'm inserting a content.js file. So if you see here there's a JS file 466.51s - 472.27s | here which is simply just gonna write a text on the page. So you know if I go 472.27s - 475.99s | back to my browser and let's say you know I try to load this specific 475.99s - 485.51s | extension and if I disable it now if I try to access a website let's say 485.51s - 490.99s | google.com it loads normally but the moment I basically enable this test 490.99s - 496.31s | extension and


chunk_text:
would say the very first thing 434.51s - 439.43s | that we all should you know be careful about because content scripts are 439.43s - 444.91s | injected on the web pages. Now to see it in action I'll actually show you a live 444.91s - 449.35s | code right now to see how exactly content script can modify a given page. 449.35s - 456.63s | So it's actually pretty straightforward. This is a typical extension. You can see 456.63s - 461.59s | like there's a manifest file. I'm just naming it test extension for now and in 461.59s - 466.51s | this I'm inserting a content.js file. So if you see here there's a JS file 466.51s - 472.27s | here which is simply just gonna write a text on the page. So you know if I go 472.27s - 475.99s | back to my browser and let's say you know I try to load this specific 475.99s - 485.51s | extension and if I disable it now if I try to access a website let's say 485.51s - 490.99s | google.com it loads normally but the moment I basically enable this test 490.99s - 496.31s | extension and I try to open google.com it will get overwritten by whatever I 496.31s - 500.83s | have specified on the extension. Now this is how basically content script works 500.83s - 505.99s | like you know you have a way to inject something on the web page. Now if you see 505.99s - 511.71s | this specific extension there is no permission key here and that's what 511.71s - 515.55s | makes it more dangerous because you know I was able to modify a page but I'm not 515.55s - 520.35s | actually asking for any explicit permission other than like in what pages 520.35s - 524.43s | I'm actually inserting the content script. So it's important not just to 524.43s - 527.39s | look at the permissions key but also to look at you know in what particular 527.59s - 533.55s | pages the content script is being injected to. Now coming back one of the 533.55s - 537.83s | biggest problems that arise with this is actually remote code execution. So let's 537.83s - 541.91s | say you know I allow any of the extension to be installed on your 541.91s - 547.03s | browser and later on


chunk_text:
I try to open google.com it will get overwritten by whatever I 496.31s - 500.83s | have specified on the extension. Now this is how basically content script works 500.83s - 505.99s | like you know you have a way to inject something on the web page. Now if you see 505.99s - 511.71s | this specific extension there is no permission key here and that's what 511.71s - 515.55s | makes it more dangerous because you know I was able to modify a page but I'm not 515.55s - 520.35s | actually asking for any explicit permission other than like in what pages 520.35s - 524.43s | I'm actually inserting the content script. So it's important not just to 524.43s - 527.39s | look at the permissions key but also to look at you know in what particular 527.59s - 533.55s | pages the content script is being injected to. Now coming back one of the 533.55s - 537.83s | biggest problems that arise with this is actually remote code execution. So let's 537.83s - 541.91s | say you know I allow any of the extension to be installed on your 541.91s - 547.03s | browser and later on I'm sending some code from my server which is definitely 547.03s - 552.07s | unmonitored and I'm sending that code to execute on any random web page. Now this 552.07s - 556.67s | is quite dangerous so to avoid this lot of changes were done in the mv3 version 556.71s - 560.75s | itself. So we have these functions like eval where you know you can specify like 560.75s - 565.51s | any kind of JavaScript code and it executes on a given page. So all these 565.51s - 570.23s | things were banned in the mv3 at the same time also enforced to be blocked 570.23s - 575.63s | using the content security policy that is CSP. However there are ways to bypass 575.63s - 582.15s | this as well. So you know just to give a small example of this if I go back and 582.35s - 587.35s | look at the very same extension which I just showed you and if I try this with 587.35s - 591.79s | you know just an eval. Now this is how eval looks like you know you're sending 591.79s - 596.11s | a string and this string is going to


chunk_text:
I'm sending some code from my server which is definitely 547.03s - 552.07s | unmonitored and I'm sending that code to execute on any random web page. Now this 552.07s - 556.67s | is quite dangerous so to avoid this lot of changes were done in the mv3 version 556.71s - 560.75s | itself. So we have these functions like eval where you know you can specify like 560.75s - 565.51s | any kind of JavaScript code and it executes on a given page. So all these 565.51s - 570.23s | things were banned in the mv3 at the same time also enforced to be blocked 570.23s - 575.63s | using the content security policy that is CSP. However there are ways to bypass 575.63s - 582.15s | this as well. So you know just to give a small example of this if I go back and 582.35s - 587.35s | look at the very same extension which I just showed you and if I try this with 587.35s - 591.79s | you know just an eval. Now this is how eval looks like you know you're sending 591.79s - 596.11s | a string and this string is going to execute on the page. If I go back to my 596.11s - 602.07s | browser and I try to you know reload my extension and if I refresh now the page 602.07s - 607.15s | will load normally. Reason being that it has been blocked because of 607.15s - 613.51s | a CSP policy. However JavaScript is a funny language so you can actually 613.51s - 618.95s | access the object the functions like normal arrays or objects as well. So if I 618.95s - 622.71s | go down and I write simply something like this like you know so I specify my 622.71s - 628.47s | right within the square brackets and the argument in a simple string then it 628.47s - 633.27s | actually work as well. So if I go back and I you know basically try to refresh 633.27s - 638.03s | my extension and if I go back this will again show me back hello from extension. 638.03s - 643.27s | Now as you can see all these two things were simple strings and this can 643.27s - 647.39s | actually be leveraged to further send those strings from an external 647.39s - 652.53s |


chunk_text:
execute on the page. If I go back to my 596.11s - 602.07s | browser and I try to you know reload my extension and if I refresh now the page 602.07s - 607.15s | will load normally. Reason being that it has been blocked because of 607.15s - 613.51s | a CSP policy. However JavaScript is a funny language so you can actually 613.51s - 618.95s | access the object the functions like normal arrays or objects as well. So if I 618.95s - 622.71s | go down and I write simply something like this like you know so I specify my 622.71s - 628.47s | right within the square brackets and the argument in a simple string then it 628.47s - 633.27s | actually work as well. So if I go back and I you know basically try to refresh 633.27s - 638.03s | my extension and if I go back this will again show me back hello from extension. 638.03s - 643.27s | Now as you can see all these two things were simple strings and this can 643.27s - 647.39s | actually be leveraged to further send those strings from an external 647.39s - 652.53s | server as well. And this can also be used to in fact you know change the URL as 652.53s - 656.79s | well. So if you see the snippet here this will simply change the location of a 656.79s - 663.07s | given page. Now you know the same thing can be extended by connecting this to a 663.07s - 668.55s | server as well. So if you see here I have a simple Flask server which just 668.55s - 672.29s | returns two things. One is a function name and the other one is just an 672.29s - 680.25s | argument. So if I run this server right now so I have my Flask server running 680.25s - 687.01s | and I'll simply go back and I will load the other extension which has an 687.01s - 701.50s | integration with this API. Now if I try to access the page it still works and 701.50s - 705.70s | you can see a hello from server is written there. Reason being that if you 705.70s - 710.70s | look at the code of this extension it is making an API call, getting what the 710.70s - 715.00s | function


chunk_text:
server as well. And this can also be used to in fact you know change the URL as 652.53s - 656.79s | well. So if you see the snippet here this will simply change the location of a 656.79s - 663.07s | given page. Now you know the same thing can be extended by connecting this to a 663.07s - 668.55s | server as well. So if you see here I have a simple Flask server which just 668.55s - 672.29s | returns two things. One is a function name and the other one is just an 672.29s - 680.25s | argument. So if I run this server right now so I have my Flask server running 680.25s - 687.01s | and I'll simply go back and I will load the other extension which has an 687.01s - 701.50s | integration with this API. Now if I try to access the page it still works and 701.50s - 705.70s | you can see a hello from server is written there. Reason being that if you 705.70s - 710.70s | look at the code of this extension it is making an API call, getting what the 710.70s - 715.00s | function name is, getting what the argument is and simply using it to 715.00s - 720.48s | change the content of the page. 720.48s - 725.84s | Okay so I think you know given the talk time is limited we'll start with some 725.84s - 731.16s | demo videos as well. I think we have 10 minutes on the clock. 10 minutes? Okay yeah 731.20s - 736.12s | So the very first demo you want to show is I mean there are a bunch of them is 736.12s - 740.92s | social engineering where you could literally once you install an extension 740.92s - 745.40s | it could pretend to go ahead and let's say drop a zoom update right because 745.40s - 749.40s | it's so common that you try to access zoom and it says go update and Shoury 749.40s - 756.74s | will show you that. So we can see here that you know I'm simply opening the 756.74s - 761.10s | zoom website and it looks like you know a normal zoom website how it's supposed 761.10s - 765.78s | to be. If I go ahead and enable this malicious extension it's simply using a 765.78s -


chunk_text:
name is, getting what the argument is and simply using it to 715.00s - 720.48s | change the content of the page. 720.48s - 725.84s | Okay so I think you know given the talk time is limited we'll start with some 725.84s - 731.16s | demo videos as well. I think we have 10 minutes on the clock. 10 minutes? Okay yeah 731.20s - 736.12s | So the very first demo you want to show is I mean there are a bunch of them is 736.12s - 740.92s | social engineering where you could literally once you install an extension 740.92s - 745.40s | it could pretend to go ahead and let's say drop a zoom update right because 745.40s - 749.40s | it's so common that you try to access zoom and it says go update and Shoury 749.40s - 756.74s | will show you that. So we can see here that you know I'm simply opening the 756.74s - 761.10s | zoom website and it looks like you know a normal zoom website how it's supposed 761.10s - 765.78s | to be. If I go ahead and enable this malicious extension it's simply using a 765.78s - 770.50s | content script to add another object on that specific page which says hey there's 770.50s - 775.14s | an update now. At the moment you click on this you download a file and if you see 775.14s - 779.46s | here the download we are triggering is actually a blog based download. So even 779.46s - 783.02s | if you go to the downloads and you look at the URL you will see it as you know 783.02s - 789.77s | it's coming from zoom.us but actually it was shipped with the extension. Now if we 789.77s - 794.49s | simply open that executable it might affect your computer. The the key thing 794.49s - 798.77s | to realize over here is extensions have the power to run in the context of 798.77s - 802.81s | whichever page that they're injecting into. So if you install a random 802.81s - 807.65s | extension which says it's a color picker and then you open zoom.us at that point 807.65s - 812.81s | it can actually inject into that page and anything which happens from the 812.81s - 817.01s | browser's perspective is in the context of the page. So


chunk_text:
770.50s | content script to add another object on that specific page which says hey there's 770.50s - 775.14s | an update now. At the moment you click on this you download a file and if you see 775.14s - 779.46s | here the download we are triggering is actually a blog based download. So even 779.46s - 783.02s | if you go to the downloads and you look at the URL you will see it as you know 783.02s - 789.77s | it's coming from zoom.us but actually it was shipped with the extension. Now if we 789.77s - 794.49s | simply open that executable it might affect your computer. The the key thing 794.49s - 798.77s | to realize over here is extensions have the power to run in the context of 798.77s - 802.81s | whichever page that they're injecting into. So if you install a random 802.81s - 807.65s | extension which says it's a color picker and then you open zoom.us at that point 807.65s - 812.81s | it can actually inject into that page and anything which happens from the 812.81s - 817.01s | browser's perspective is in the context of the page. So if a file download is 817.01s - 822.65s | triggered you feel it's probably from you know the actual zoom website. Now one 822.65s - 827.85s | interesting demo he'll actually show you is something live. Yeah. Okay and while 827.85s - 833.45s | he's he's getting that ready. Imagine you have an extension and now you're doing a 833.45s - 838.09s | video call right. How many of you end up using the browser version of zoom and 838.09s - 843.29s | Google Meets and whatnot. Now you might actually imagine that when a Google Meet 843.41s - 848.21s | call is actually happening nothing else can monitor. So we'll show you a demo of 848.21s - 854.05s | an extension which we've created which while a live call is happening it can go 854.05s - 859.73s | ahead do a side monitoring and literally get the entire feed of the call live and 859.73s - 864.41s | send it wherever. With zero indication in the Google Meet session that something 864.41s - 870.21s | is intercepting or monitoring this. So as you can see here you know I'm basically 870.33s - 883.30s | gonna open Google Meet. We


chunk_text:
if a file download is 817.01s - 822.65s | triggered you feel it's probably from you know the actual zoom website. Now one 822.65s - 827.85s | interesting demo he'll actually show you is something live. Yeah. Okay and while 827.85s - 833.45s | he's he's getting that ready. Imagine you have an extension and now you're doing a 833.45s - 838.09s | video call right. How many of you end up using the browser version of zoom and 838.09s - 843.29s | Google Meets and whatnot. Now you might actually imagine that when a Google Meet 843.41s - 848.21s | call is actually happening nothing else can monitor. So we'll show you a demo of 848.21s - 854.05s | an extension which we've created which while a live call is happening it can go 854.05s - 859.73s | ahead do a side monitoring and literally get the entire feed of the call live and 859.73s - 864.41s | send it wherever. With zero indication in the Google Meet session that something 864.41s - 870.21s | is intercepting or monitoring this. So as you can see here you know I'm basically 870.33s - 883.30s | gonna open Google Meet. We hope the internet connection works or else we'll 883.30s - 886.10s | show you a very quick video. Okay we have five minutes on the clock while he's 886.10s - 892.22s | getting it. The key thing to realize is you can actually host a Chromium 892.22s - 897.66s | extension or an Edge extension. Much of the times there is barely any security 897.66s - 902.26s | checks which are really happening. So if you see a featured extension it isn't 902.26s - 905.82s | that you know Google's team has looked at the code run a bunch of tests 905.82s - 910.54s | unbelievably even in the enterprise version some of the basic stuff that 910.54s - 916.18s | they end up showing doesn't really look at what that code could do. Now here's 916.18s - 921.18s | the worst part. We all fear download and execs right because you run the stub get 921.18s - 925.30s | something that's something extensions can do right off the box. Is the 925.30s - 929.26s | extension just loads looks innocent but it can actually bring back foreign code 929.26s - 937.77s | and run it. So if you see


chunk_text:
hope the internet connection works or else we'll 883.30s - 886.10s | show you a very quick video. Okay we have five minutes on the clock while he's 886.10s - 892.22s | getting it. The key thing to realize is you can actually host a Chromium 892.22s - 897.66s | extension or an Edge extension. Much of the times there is barely any security 897.66s - 902.26s | checks which are really happening. So if you see a featured extension it isn't 902.26s - 905.82s | that you know Google's team has looked at the code run a bunch of tests 905.82s - 910.54s | unbelievably even in the enterprise version some of the basic stuff that 910.54s - 916.18s | they end up showing doesn't really look at what that code could do. Now here's 916.18s - 921.18s | the worst part. We all fear download and execs right because you run the stub get 921.18s - 925.30s | something that's something extensions can do right off the box. Is the 925.30s - 929.26s | extension just loads looks innocent but it can actually bring back foreign code 929.26s - 937.77s | and run it. So if you see here we are starting a new Google Meet session. Now 937.77s - 942.65s | the malicious extension is installed. So we're gonna simply open you know another 942.65s - 946.49s | window and open an extension page to see what's actually seeing and you can see 946.49s - 950.49s | like you know the live feed from that particular meet page is being transferred 950.49s - 955.17s | to the other page. And that shows a lot because you know having a simple a 955.17s - 959.49s | simple host permission it was not even a specified permission you were actually 959.49s - 965.69s | able to steal a live stream from a given meet page. So the key thing to realize is 965.69s - 970.61s | you know extensions have superpowers just like you know an endpoint security 970.61s - 974.05s | you end up installing something in kernel mode and that has one privilege 974.05s - 978.93s | level above what user space processes have. Similarly extensions are sitting 978.93s - 983.77s | one level above you know the page itself. Yeah we'll also look at just two quick 983.77s - 989.13s | demos right now. Another one


chunk_text:
here we are starting a new Google Meet session. Now 937.77s - 942.65s | the malicious extension is installed. So we're gonna simply open you know another 942.65s - 946.49s | window and open an extension page to see what's actually seeing and you can see 946.49s - 950.49s | like you know the live feed from that particular meet page is being transferred 950.49s - 955.17s | to the other page. And that shows a lot because you know having a simple a 955.17s - 959.49s | simple host permission it was not even a specified permission you were actually 959.49s - 965.69s | able to steal a live stream from a given meet page. So the key thing to realize is 965.69s - 970.61s | you know extensions have superpowers just like you know an endpoint security 970.61s - 974.05s | you end up installing something in kernel mode and that has one privilege 974.05s - 978.93s | level above what user space processes have. Similarly extensions are sitting 978.93s - 983.77s | one level above you know the page itself. Yeah we'll also look at just two quick 983.77s - 989.13s | demos right now. Another one is called silent account hijacking. So in this what 989.13s - 993.57s | I'm gonna do is I'll simply have a malicious extension installed and what 993.57s - 997.53s | it's gonna do is that in the background it's gonna log into your GitHub account 997.53s - 1001.69s | because extensions have a highly privileged so they have access to all 1001.69s - 1006.49s | HTTP only cookies as well. And if you're doing it from your service worker that 1006.49s - 1010.29s | is from your background you don't even have to ask for explicit permission for 1010.29s - 1013.85s | accessing cookie as well. If you're simply making a fetch call instead of 1013.85s - 1019.89s | you know just getting the cookie value. So in this demo we can see that we have 1019.89s - 1027.00s | a demo GitHub account and I have a private repo here. In this private repo 1027.00s - 1032.56s | if we go to the settings and look at the collaborators right now there are no 1032.56s - 1037.60s | additional collaborators in this repo. Now what I'll do is I'll simply go and 1037.60s - 1045.21s | start the malicious


chunk_text:
is called silent account hijacking. So in this what 989.13s - 993.57s | I'm gonna do is I'll simply have a malicious extension installed and what 993.57s - 997.53s | it's gonna do is that in the background it's gonna log into your GitHub account 997.53s - 1001.69s | because extensions have a highly privileged so they have access to all 1001.69s - 1006.49s | HTTP only cookies as well. And if you're doing it from your service worker that 1006.49s - 1010.29s | is from your background you don't even have to ask for explicit permission for 1010.29s - 1013.85s | accessing cookie as well. If you're simply making a fetch call instead of 1013.85s - 1019.89s | you know just getting the cookie value. So in this demo we can see that we have 1019.89s - 1027.00s | a demo GitHub account and I have a private repo here. In this private repo 1027.00s - 1032.56s | if we go to the settings and look at the collaborators right now there are no 1032.56s - 1037.60s | additional collaborators in this repo. Now what I'll do is I'll simply go and 1037.60s - 1045.21s | start the malicious extension. Now if you see nothing really happened and for a 1045.21s - 1050.33s | very small moment a small tab just open and closed automatically. Now if I go 1050.33s - 1057.42s | back and check that specific repo back again go to the settings and see 1057.42s - 1063.04s | who all are the collaborators. Then I will see that there's one more 1063.04s - 1068.64s | collaborator added here now. So this just shows you know how how sneaky they can 1068.64s - 1073.76s | be and they can actually also take over your private repos. And and the best way 1073.76s - 1077.20s | to think about it is think about any form of browser automation with things 1077.20s - 1081.44s | like Selenium which some of you may have done. With an extension it can just do 1081.44s - 1085.76s | literally everything in the background, act on your behalf, go through complex 1085.76s - 1090.60s | workflows, anything the you know author would have probably baked into it. And 1090.60s - 1095.16s | some of these can just download at payloads and happen at runtime. Doesn't 1095.16s - 1101.20s | need to be at


chunk_text:
extension. Now if you see nothing really happened and for a 1045.21s - 1050.33s | very small moment a small tab just open and closed automatically. Now if I go 1050.33s - 1057.42s | back and check that specific repo back again go to the settings and see 1057.42s - 1063.04s | who all are the collaborators. Then I will see that there's one more 1063.04s - 1068.64s | collaborator added here now. So this just shows you know how how sneaky they can 1068.64s - 1073.76s | be and they can actually also take over your private repos. And and the best way 1073.76s - 1077.20s | to think about it is think about any form of browser automation with things 1077.20s - 1081.44s | like Selenium which some of you may have done. With an extension it can just do 1081.44s - 1085.76s | literally everything in the background, act on your behalf, go through complex 1085.76s - 1090.60s | workflows, anything the you know author would have probably baked into it. And 1090.60s - 1095.16s | some of these can just download at payloads and happen at runtime. Doesn't 1095.16s - 1101.20s | need to be at you know extension compile or upload time to the Chrome store. Yeah 1101.20s - 1105.16s | now we look at one more which is interference with password managers. So 1105.16s - 1109.32s | many people often use extensions for you know using saving their password and 1109.32s - 1113.28s | using them. So actually other malicious extension can interfere with that as 1113.28s - 1119.60s | well. So in this example we can see that a user is using one of the password 1119.60s - 1124.28s | manager vendors. And if you go to let's say a random SaaS application where you 1124.28s - 1128.96s | were already using that specific password manager. We can see that the 1128.96s - 1133.60s | password manager works and a person is you know able to log in normally how 1133.60s - 1140.08s | it's supposed to be. However the moment if we go back and enable the malicious 1140.08s - 1144.96s | extension, what this extension will do is that it will override what the default 1144.96s - 1149.60s | action of clicking on that icon was. So if we can see here it's still the 1149.60s - 1154.44s | password managers


chunk_text:
you know extension compile or upload time to the Chrome store. Yeah 1101.20s - 1105.16s | now we look at one more which is interference with password managers. So 1105.16s - 1109.32s | many people often use extensions for you know using saving their password and 1109.32s - 1113.28s | using them. So actually other malicious extension can interfere with that as 1113.28s - 1119.60s | well. So in this example we can see that a user is using one of the password 1119.60s - 1124.28s | manager vendors. And if you go to let's say a random SaaS application where you 1124.28s - 1128.96s | were already using that specific password manager. We can see that the 1128.96s - 1133.60s | password manager works and a person is you know able to log in normally how 1133.60s - 1140.08s | it's supposed to be. However the moment if we go back and enable the malicious 1140.08s - 1144.96s | extension, what this extension will do is that it will override what the default 1144.96s - 1149.60s | action of clicking on that icon was. So if we can see here it's still the 1149.60s - 1154.44s | password managers icon but the moment I click on this it will take me to another 1154.44s - 1159.48s | login page. So for the test purpose and for demonstration we have just named our 1159.48s - 1163.96s | extension with a malware word so that you know that this page is actually you 1163.96s - 1168.36s | know a fake one. But you know looking at the URL you can't really figure that out. 1168.36s - 1172.62s | So here a person will then you know saying that they might have logged out 1172.62s - 1176.76s | of their password manager and they will end up ending the credentials. And the 1176.76s - 1182.92s | moment they do that they will get phished. So you run out of time. I think 1182.92s - 1186.56s | if you're interested all the source code everything we have here you can go to 1186.56s - 1192.08s | sqrx.com. We put all of that up. We also put up a much more detailed video. We 1192.08s - 1198.28s | only got 20 minutes. We have a lot of research. Now the key summary is even 1198.28s - 1202.08s | known extensions on the Chrome store


chunk_text:
icon but the moment I click on this it will take me to another 1154.44s - 1159.48s | login page. So for the test purpose and for demonstration we have just named our 1159.48s - 1163.96s | extension with a malware word so that you know that this page is actually you 1163.96s - 1168.36s | know a fake one. But you know looking at the URL you can't really figure that out. 1168.36s - 1172.62s | So here a person will then you know saying that they might have logged out 1172.62s - 1176.76s | of their password manager and they will end up ending the credentials. And the 1176.76s - 1182.92s | moment they do that they will get phished. So you run out of time. I think 1182.92s - 1186.56s | if you're interested all the source code everything we have here you can go to 1186.56s - 1192.08s | sqrx.com. We put all of that up. We also put up a much more detailed video. We 1192.08s - 1198.28s | only got 20 minutes. We have a lot of research. Now the key summary is even 1198.28s - 1202.08s | known extensions on the Chrome store get bought out all the time. And what 1202.08s - 1205.16s | attackers have started doing is if an extension has a hundred thousand 1205.16s - 1209.20s | installs they just offer ten thousand dollars to the author and they can 1209.20s - 1214.96s | transfer over ownership. And that means you know extensions you trusted could 1214.96s - 1219.08s | now automatically be untrusted. And a bunch of other supply chain attacks 1219.08s - 1222.56s | which can kind of happen in the context of the Chrome store right. Thank you so 1222.56s - 1228.64s | much and yeah please visit sqrx.com for more. Thank you.


chunk_text:
get bought out all the time. And what 1202.08s - 1205.16s | attackers have started doing is if an extension has a hundred thousand 1205.16s - 1209.20s | installs they just offer ten thousand dollars to the author and they can 1209.20s - 1214.96s | transfer over ownership. And that means you know extensions you trusted could 1214.96s - 1219.08s | now automatically be untrusted. And a bunch of other supply chain attacks 1219.08s - 1222.56s | which can kind of happen in the context of the Chrome store right. Thank you so 1222.56s - 1228.64s | much and yeah please visit sqrx.com for more. Thank you.


chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=Cxrwpij2MSQ", "title": "DEF CON 32 - SBOMs the Hard Way Hacking Bob the Minion - Larry Pesce", "description": "This presentation delves into the intricate process of generating a Software Bill of Materials (SBOM) for the Bob the Minions WiFi router by Davolink\u2014a device whose firmware isn't publicly available. Traditional SBOM creation methods rely on readily accessible firmware, but Davolink's restricted release policies necessitated an unconventional approach. This talk covers the step-by-step journey of hardware disassembly, firmware extraction via SPI flash and JTAG/SWD interfaces, and the tools and techniques employed. Finally, we'll demonstrate how the recovered firmware is used to generate a comprehensive SBOM, highlighting any security vulnerabilities discovered and reported to the vendor. This session aims to provide attendees with practical insights into overcoming SBOM generation challenges in the IoT domain through hands-on hardware hacking, and leveraging the firmware and SBOMs for vulnerability discovery, as well as security improvement.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1254, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 4.08s | This text was transcribed using whisper model: large-v2 Alright, let's get started. We're running a little late. So I'm going to try not to talk too fast. 4.72s - 11.44s | But I got to make up some time here. So welcome to s bombs the hard way. For my adventures in 11.44s - 17.28s | hacking Bob the Minion. I am the product security research and analysis director and services team 17.28s - 24.24s | lead at a fun company called finite state. And we do all sorts of fun stuff with s bombs, and a 24.24s - 27.84s | whole bunch of other stuff. But I'm not here to talk to you about our product or any of that I'm 27.84s - 36.96s | talking about my adventure with Bob. Alright, so who am I? I am a reformed penetration tester 36.96s - 43.36s | and reformed crossed out. I left doing penetration testing to go do some thought leadership and some 43.36s - 48.72s | other hardware hacking and IoT bunch of stuff. And three months into the job, my boss came to 48.72s - 51.84s | me and said, Hey, we have a customer that wants to throw a lot of money at us to do pen tests. 51.84s - 57.28s | Can you do it? Are you signing my paycheck? Yes, I can do that. So I am a


chunk_text:
to make up some time here. So welcome to s bombs the hard way. For my adventures in 11.44s - 17.28s | hacking Bob the Minion. I am the product security research and analysis director and services team 17.28s - 24.24s | lead at a fun company called finite state. And we do all sorts of fun stuff with s bombs, and a 24.24s - 27.84s | whole bunch of other stuff. But I'm not here to talk to you about our product or any of that I'm 27.84s - 36.96s | talking about my adventure with Bob. Alright, so who am I? I am a reformed penetration tester 36.96s - 43.36s | and reformed crossed out. I left doing penetration testing to go do some thought leadership and some 43.36s - 48.72s | other hardware hacking and IoT bunch of stuff. And three months into the job, my boss came to 48.72s - 51.84s | me and said, Hey, we have a customer that wants to throw a lot of money at us to do pen tests. 51.84s - 57.28s | Can you do it? Are you signing my paycheck? Yes, I can do that. So I am a reformed penetration 57.28s - 65.12s | tester back doing penetration testing back in ot ICS, IoT, and lots of hardware. I'm also the 65.12s - 71.36s | co host of Paul's security weekly. We're in our 19th year soon to go into our 20th year of a 71.36s - 78.64s | weekly security podcast. How many of your listeners? Go listen. Right. Awesome. Thank you. 79.76s - 86.48s | I do a lot of work with sans. I'm both an instructor and a course author around wireless 86.48s - 93.60s | hacking and IoT hacking irony, the ironically, I'm a recent convert to the Oxford comma, because 93.60s - 99.44s | commas save lives, folks. Okay. And someone said, Why are you been using that Oxford comma for a 99.44s - 103.68s | really long time? So you're really not a recent convert. So but I'm going to keep it. So I'll add 103.68s - 110.40s | that. I also enjoy long walks on the beach weighed down by my ham radio. Yeah, prepping for the 110.40s - 116.16s | zombie apocalypse. Okay. Alright, so let's get into some fun, fun bits here. So we're going to 116.16s - 125.12s


chunk_text:
reformed penetration 57.28s - 65.12s | tester back doing penetration testing back in ot ICS, IoT, and lots of hardware. I'm also the 65.12s - 71.36s | co host of Paul's security weekly. We're in our 19th year soon to go into our 20th year of a 71.36s - 78.64s | weekly security podcast. How many of your listeners? Go listen. Right. Awesome. Thank you. 79.76s - 86.48s | I do a lot of work with sans. I'm both an instructor and a course author around wireless 86.48s - 93.60s | hacking and IoT hacking irony, the ironically, I'm a recent convert to the Oxford comma, because 93.60s - 99.44s | commas save lives, folks. Okay. And someone said, Why are you been using that Oxford comma for a 99.44s - 103.68s | really long time? So you're really not a recent convert. So but I'm going to keep it. So I'll add 103.68s - 110.40s | that. I also enjoy long walks on the beach weighed down by my ham radio. Yeah, prepping for the 110.40s - 116.16s | zombie apocalypse. Okay. Alright, so let's get into some fun, fun bits here. So we're going to 116.16s - 125.12s | hack Bob the minion. So on every one of my slides, except this one features AI generated art 126.00s - 133.36s | with minions. So let's get into character a little bit here. And I took this image of my headshot 133.36s - 141.28s | and fed fed it to mid journey so that I could become an evil, despicable me villain. So 142.24s - 149.20s | here we go. This is some of this art is terrifying. And I kept the ones that were insane just because 149.20s - 156.48s | insanity. I didn't know yesterday that there was an AI generated art competition. My I would have 156.48s - 163.28s | been good at that. Maybe. So I had lots of fun. I think I put more time into the art than the actual 163.28s - 167.20s | words on this on these slides. Don't get me wrong this. I think the presentation is really good. 167.20s - 173.12s | You're going to get something from it. Alright, so a little bit of background. I work for company, 173.12s - 179.04s | we do s bombs and a whole bunch of other stuff. But I really think


chunk_text:
| hack Bob the minion. So on every one of my slides, except this one features AI generated art 126.00s - 133.36s | with minions. So let's get into character a little bit here. And I took this image of my headshot 133.36s - 141.28s | and fed fed it to mid journey so that I could become an evil, despicable me villain. So 142.24s - 149.20s | here we go. This is some of this art is terrifying. And I kept the ones that were insane just because 149.20s - 156.48s | insanity. I didn't know yesterday that there was an AI generated art competition. My I would have 156.48s - 163.28s | been good at that. Maybe. So I had lots of fun. I think I put more time into the art than the actual 163.28s - 167.20s | words on this on these slides. Don't get me wrong this. I think the presentation is really good. 167.20s - 173.12s | You're going to get something from it. Alright, so a little bit of background. I work for company, 173.12s - 179.04s | we do s bombs and a whole bunch of other stuff. But I really think that s bombs are fun. And that 179.04s - 186.08s | was sort of my end goal was to get a software bill of materials for Bob the minion. So a little bit 186.08s - 191.52s | of background. software bill of materials is an inventory of all your software, their versions, 191.68s - 195.68s | and who wrote it, where it came from, as part of your product security, 196.56s - 202.08s | sorry, your software supply chain, and can help you do some security. And we're going to start 202.08s - 208.32s | seeing somewhere around 2025, that every product that connected product that you buy, either at 208.32s - 213.60s | your company or for your personal consumption, will have an s mom associated with it, and will 213.60s - 217.76s | be available as part of some of its certification process and being able to be sold into the 217.76s - 223.76s | marketplace. Specifically, addressing some of those here in the United States are executive 223.76s - 232.56s | order 14 208. For a lot of stuff going into government and into consumer stuff. 524 B for 232.56s - 239.36s | the FDA. So any


chunk_text:
that s bombs are fun. And that 179.04s - 186.08s | was sort of my end goal was to get a software bill of materials for Bob the minion. So a little bit 186.08s - 191.52s | of background. software bill of materials is an inventory of all your software, their versions, 191.68s - 195.68s | and who wrote it, where it came from, as part of your product security, 196.56s - 202.08s | sorry, your software supply chain, and can help you do some security. And we're going to start 202.08s - 208.32s | seeing somewhere around 2025, that every product that connected product that you buy, either at 208.32s - 213.60s | your company or for your personal consumption, will have an s mom associated with it, and will 213.60s - 217.76s | be available as part of some of its certification process and being able to be sold into the 217.76s - 223.76s | marketplace. Specifically, addressing some of those here in the United States are executive 223.76s - 232.56s | order 14 208. For a lot of stuff going into government and into consumer stuff. 524 B for 232.56s - 239.36s | the FDA. So any medical device that is being submitted to the FDA for use on humans requires 239.36s - 244.56s | that an s bomb as part of their submission process and a plan for securing their devices over time 244.56s - 250.32s | and managing vulnerabilities. And a voluntary program managed by the FCC here in the US, 250.32s - 256.56s | the cyber Trustmark program, it's kind of like a consumer protection thing so that as a voluntary 256.56s - 262.56s | program, that companies can submit their devices to the cyber Trustmark program, they can get a 262.56s - 267.76s | stamp on the box, and that a consumer can have a high degree of confidence that there is some 267.76s - 273.04s | security in this product baked in, and that they are monitoring for vulnerabilities over time and 273.04s - 277.12s | issuing some updates with a little barcode that links to a website that gives them an overall 277.12s - 282.08s | security store score. So while you're standing at Best Buy, you can scan this barcode and go, 282.08s - 287.76s | oh, this routers got a three for security, and that one's got a nine,


chunk_text:
medical device that is being submitted to the FDA for use on humans requires 239.36s - 244.56s | that an s bomb as part of their submission process and a plan for securing their devices over time 244.56s - 250.32s | and managing vulnerabilities. And a voluntary program managed by the FCC here in the US, 250.32s - 256.56s | the cyber Trustmark program, it's kind of like a consumer protection thing so that as a voluntary 256.56s - 262.56s | program, that companies can submit their devices to the cyber Trustmark program, they can get a 262.56s - 267.76s | stamp on the box, and that a consumer can have a high degree of confidence that there is some 267.76s - 273.04s | security in this product baked in, and that they are monitoring for vulnerabilities over time and 273.04s - 277.12s | issuing some updates with a little barcode that links to a website that gives them an overall 277.12s - 282.08s | security store score. So while you're standing at Best Buy, you can scan this barcode and go, 282.08s - 287.76s | oh, this routers got a three for security, and that one's got a nine, nine being better. Yeah, 287.76s - 294.24s | I'm going to buy the one that's a nine. Cool. The one that is absolutely both fascinating and 294.24s - 303.12s | terrifying, in my opinion, is the EU Cyber Resiliency Act, or the EU CRA. EU CRA, their 303.12s - 308.72s | intent in the EU is that they do not want any connected products being sold into the marketplace 308.72s - 314.96s | that has any vulnerabilities. And they also need to have a process for managing vulnerabilities 314.96s - 319.44s | and patching and monitoring and all this type of stuff. And an s bomb will be included as part of 319.44s - 325.60s | that process. And they are significantly more stringent, and this is not a voluntary program. 326.32s - 330.96s | It's mandatory if you want to sell into Europe, and companies that sell into the US also sell 330.96s - 338.00s | into Europe, same device. So many manufacturers are going to have to submit to EU CRA, which means 338.80s - 344.56s | everywhere on the planet will largely benefit from those regulations. Overall, some general 344.56s - 350.08s | benefits outside of any


chunk_text:
nine being better. Yeah, 287.76s - 294.24s | I'm going to buy the one that's a nine. Cool. The one that is absolutely both fascinating and 294.24s - 303.12s | terrifying, in my opinion, is the EU Cyber Resiliency Act, or the EU CRA. EU CRA, their 303.12s - 308.72s | intent in the EU is that they do not want any connected products being sold into the marketplace 308.72s - 314.96s | that has any vulnerabilities. And they also need to have a process for managing vulnerabilities 314.96s - 319.44s | and patching and monitoring and all this type of stuff. And an s bomb will be included as part of 319.44s - 325.60s | that process. And they are significantly more stringent, and this is not a voluntary program. 326.32s - 330.96s | It's mandatory if you want to sell into Europe, and companies that sell into the US also sell 330.96s - 338.00s | into Europe, same device. So many manufacturers are going to have to submit to EU CRA, which means 338.80s - 344.56s | everywhere on the planet will largely benefit from those regulations. Overall, some general 344.56s - 350.08s | benefits outside of any of the regulatory stuff for software bill materials, they give us that 350.08s - 356.08s | software inventory list for both both first and third party stuff that you're developing and stuff 356.08s - 361.12s | that you've pulled in as part of your development process, and even into potential final firmware 361.12s - 366.72s | that you're delivering to your customers or you are consuming. With that list of those components 366.72s - 372.56s | and their versions, you can perform some enrichment on those find out whether there are vulnerabilities 372.56s - 378.16s | associated with those products so that you can then attempt to perform some risk mitigation and 378.16s - 383.44s | observe that some of that risk exists and take appropriate actions. And I'll also argue that 383.44s - 389.20s | you should be operationalizing your s bombs as either a consumer or a manufacturer, that you 389.20s - 394.80s | should take the software bill of materials and use them as part of your security process and your 394.80s - 400.24s | penetration testing process. Because knowing the components may give you an opportunity 400.88s - 405.44s | to improve that security knowing that


chunk_text:
of the regulatory stuff for software bill materials, they give us that 350.08s - 356.08s | software inventory list for both both first and third party stuff that you're developing and stuff 356.08s - 361.12s | that you've pulled in as part of your development process, and even into potential final firmware 361.12s - 366.72s | that you're delivering to your customers or you are consuming. With that list of those components 366.72s - 372.56s | and their versions, you can perform some enrichment on those find out whether there are vulnerabilities 372.56s - 378.16s | associated with those products so that you can then attempt to perform some risk mitigation and 378.16s - 383.44s | observe that some of that risk exists and take appropriate actions. And I'll also argue that 383.44s - 389.20s | you should be operationalizing your s bombs as either a consumer or a manufacturer, that you 389.20s - 394.80s | should take the software bill of materials and use them as part of your security process and your 394.80s - 400.24s | penetration testing process. Because knowing the components may give you an opportunity 400.88s - 405.44s | to improve that security knowing that their vulnerabilities exist. And that was the subject 405.44s - 411.20s | of my talk that I did at RSA entitled s bombs for evil. So you can go check that out as well. So we 411.20s - 418.72s | won't go too far into that. So as part of my day job, and as part of being a reformed unreformed 418.72s - 425.28s | penetration tester, I really love software bill of materials. And some of the fun stuff that I get to 425.28s - 434.80s | do at work is go find some random device, get the firmware and bacon, get an s bomb for it just to 434.80s - 439.04s | see what they're see if we can do some accurate detection helps build build some of the things 439.04s - 443.52s | that we're detecting for helps improve our product. And quite honestly, I think it's really fun and 443.52s - 449.52s | find all sorts of sort of scary things. So getting that firmware doing some extraction, verifying that 449.52s - 454.96s | the extraction is accurate for our product, figuring out all new things, going back and 454.96s - 461.12s | reinverse engineering some


chunk_text:
their vulnerabilities exist. And that was the subject 405.44s - 411.20s | of my talk that I did at RSA entitled s bombs for evil. So you can go check that out as well. So we 411.20s - 418.72s | won't go too far into that. So as part of my day job, and as part of being a reformed unreformed 418.72s - 425.28s | penetration tester, I really love software bill of materials. And some of the fun stuff that I get to 425.28s - 434.80s | do at work is go find some random device, get the firmware and bacon, get an s bomb for it just to 434.80s - 439.04s | see what they're see if we can do some accurate detection helps build build some of the things 439.04s - 443.52s | that we're detecting for helps improve our product. And quite honestly, I think it's really fun and 443.52s - 449.52s | find all sorts of sort of scary things. So getting that firmware doing some extraction, verifying that 449.52s - 454.96s | the extraction is accurate for our product, figuring out all new things, going back and 454.96s - 461.12s | reinverse engineering some binaries to figure out the type of code and the code vulnerables. 461.92s - 466.08s | And as a result of that s bomb being able to discover that there's a product with a firmware 466.08s - 472.16s | that has some known vulnerabilities and known is here in air quotes, and that it's known to the 472.16s - 477.28s | community that these components have vulnerabilities, but maybe as a consumer, or maybe even 477.28s - 481.68s | as a manufacturer, you may not realize that these components have vulnerabilities, and they're using 481.68s - 487.52s | them in your, your products. So occasionally, we get to find some some oday and some other 487.52s - 493.92s | occasional hackery and discovering some surprises. And we may have some surprises here with Bob the 493.92s - 500.88s | minion. And arguably, it might not be all that surprising. After all, ultimately, I think the end 500.88s - 507.12s | result is so much fun, and arguably, I get to come talk to you about it at DEF CON. So finding all of 507.12s - 517.68s | those weird products, suddenly, a wild Bob appears. And yes, AI generated Bob


chunk_text:
binaries to figure out the type of code and the code vulnerables. 461.92s - 466.08s | And as a result of that s bomb being able to discover that there's a product with a firmware 466.08s - 472.16s | that has some known vulnerabilities and known is here in air quotes, and that it's known to the 472.16s - 477.28s | community that these components have vulnerabilities, but maybe as a consumer, or maybe even 477.28s - 481.68s | as a manufacturer, you may not realize that these components have vulnerabilities, and they're using 481.68s - 487.52s | them in your, your products. So occasionally, we get to find some some oday and some other 487.52s - 493.92s | occasional hackery and discovering some surprises. And we may have some surprises here with Bob the 493.92s - 500.88s | minion. And arguably, it might not be all that surprising. After all, ultimately, I think the end 500.88s - 507.12s | result is so much fun, and arguably, I get to come talk to you about it at DEF CON. So finding all of 507.12s - 517.68s | those weird products, suddenly, a wild Bob appears. And yes, AI generated Bob in a dystopian future, 518.40s - 522.88s | and hasn't shaved in a little bit. He might look a little bit like me, I'm kind of disturbed about 522.88s - 531.60s | that. So suddenly, a wild Bob appears. So what is this Bob? This Bob is a wireless home router, 531.60s - 540.24s | the DavoLink AX1800. And it has been licensed by Universal for DavoLink to produce these 540.24s - 547.20s | wireless routers that are in the shape of Bob. It's a nice plastic fun looking case around a 547.20s - 552.64s | wireless router for your home. It's like that's pretty cool. Pretty darn good features for the 552.64s - 561.84s | price. Bob has Wi Fi six has some easy mesh support and supports WPA3. So some of our latest 561.84s - 568.32s | Wi Fi standards for for your home. Now I've been doing some hardware hacking around consumer 568.32s - 575.04s | routers for a really, really long time. And it was typical that we were seeing devices with flash 575.04s - 581.12s | and RAM of somewhere between four and eight megs, maybe 16. Yeah, we've come a long way. This 581.12s


chunk_text:
in a dystopian future, 518.40s - 522.88s | and hasn't shaved in a little bit. He might look a little bit like me, I'm kind of disturbed about 522.88s - 531.60s | that. So suddenly, a wild Bob appears. So what is this Bob? This Bob is a wireless home router, 531.60s - 540.24s | the DavoLink AX1800. And it has been licensed by Universal for DavoLink to produce these 540.24s - 547.20s | wireless routers that are in the shape of Bob. It's a nice plastic fun looking case around a 547.20s - 552.64s | wireless router for your home. It's like that's pretty cool. Pretty darn good features for the 552.64s - 561.84s | price. Bob has Wi Fi six has some easy mesh support and supports WPA3. So some of our latest 561.84s - 568.32s | Wi Fi standards for for your home. Now I've been doing some hardware hacking around consumer 568.32s - 575.04s | routers for a really, really long time. And it was typical that we were seeing devices with flash 575.04s - 581.12s | and RAM of somewhere between four and eight megs, maybe 16. Yeah, we've come a long way. This 581.12s - 590.40s | particular one has, according to the box, and Amazon 128 megs of flash, and 256 megs of RAM. 590.40s - 596.88s | That's pretty darn robust and compared to some of the stuff that I've seen in the past. But it gets 596.88s - 602.88s | better. It's not 128 megs of flash, we'll see that in a little bit here. Currently, there's no open 602.88s - 608.48s | word support. So it is their own custom firmware, big thumbs down from Mick down front here. And 608.48s - 614.08s | they run about $70 on Amazon. So I didn't feel too bad about buying one of these things and 614.08s - 621.12s | ripping Bob's head off to see what was inside. I did not purchase the Kevin variety, the Kevin 621.12s - 627.20s | variety is a little bit taller. And one of the upgrades there is it does have Wi Fi 6E. So into 627.20s - 632.48s | the six gigahertz range, but they're about double the price. And well, this is coming out of my own 632.48s - 637.68s | personal budget. So I went for the cheap one, right? I figured that


chunk_text:
- 590.40s | particular one has, according to the box, and Amazon 128 megs of flash, and 256 megs of RAM. 590.40s - 596.88s | That's pretty darn robust and compared to some of the stuff that I've seen in the past. But it gets 596.88s - 602.88s | better. It's not 128 megs of flash, we'll see that in a little bit here. Currently, there's no open 602.88s - 608.48s | word support. So it is their own custom firmware, big thumbs down from Mick down front here. And 608.48s - 614.08s | they run about $70 on Amazon. So I didn't feel too bad about buying one of these things and 614.08s - 621.12s | ripping Bob's head off to see what was inside. I did not purchase the Kevin variety, the Kevin 621.12s - 627.20s | variety is a little bit taller. And one of the upgrades there is it does have Wi Fi 6E. So into 627.20s - 632.48s | the six gigahertz range, but they're about double the price. And well, this is coming out of my own 632.48s - 637.68s | personal budget. So I went for the cheap one, right? I figured that many of the challenges will 637.68s - 645.04s | probably be the same here. Alright, so, hey, I want firmware, right for these devices. So I can 645.04s - 652.00s | generate some as moms. Let's go get the firmware. But you tip my hat a little bit here. What 652.00s - 659.84s | firmware? All right. Unfortunately, I wasn't able to easily locate firmware for the devil. Inc. x 659.84s - 667.04s | 1800. We powered the device up, used it figured out what all the interface was about logged in. 667.68s - 671.28s | Take a look at the web interface. And sure enough, in the web interface, there is a button that says 671.28s - 676.24s | check for updates. And as of two weeks ago, when I press the check for updates button, 677.44s - 682.72s | there were no updates. There is one version of firmware for this device, the one that came on it 682.72s - 688.80s | from the factory, which will be terrifying when we see this a little bit later. So there's one 688.80s - 693.92s | version firmware, it is what it is, there's no updates, which means if I


chunk_text:
many of the challenges will 637.68s - 645.04s | probably be the same here. Alright, so, hey, I want firmware, right for these devices. So I can 645.04s - 652.00s | generate some as moms. Let's go get the firmware. But you tip my hat a little bit here. What 652.00s - 659.84s | firmware? All right. Unfortunately, I wasn't able to easily locate firmware for the devil. Inc. x 659.84s - 667.04s | 1800. We powered the device up, used it figured out what all the interface was about logged in. 667.68s - 671.28s | Take a look at the web interface. And sure enough, in the web interface, there is a button that says 671.28s - 676.24s | check for updates. And as of two weeks ago, when I press the check for updates button, 677.44s - 682.72s | there were no updates. There is one version of firmware for this device, the one that came on it 682.72s - 688.80s | from the factory, which will be terrifying when we see this a little bit later. So there's one 688.80s - 693.92s | version firmware, it is what it is, there's no updates, which means if I click the update button, 693.92s - 699.84s | there's no firmware, which means I can't capture anything off the wire. They can't get anything 699.84s - 704.08s | via web inspection or TCP dump or any of that type of stuff to try to recover the firmware 704.08s - 712.24s | from network traffic. Oh, I'll go to the website. Nope, nothing on their website, you go to their 712.24s - 718.00s | support forums, you go to their support pages. Absolutely no firmware for the x 1800 available 718.00s - 724.72s | for download, there's manuals and stuff. But no firmware. So what happens if I break it by 724.72s - 728.80s | accident, I'd have no way to upload firmware to this because the manufacturer doesn't even make 728.80s - 733.76s | it available. I checked the support forums to see if I could find links somewhere in the support 733.76s - 740.64s | forums. And that was a mess. It's dabbling is a Korean company, they have two different support 740.64s - 747.84s | forums. One is in Korean, one is in English. The English has a little bit more population to it. 748.48s - 753.44s |


chunk_text:
click the update button, 693.92s - 699.84s | there's no firmware, which means I can't capture anything off the wire. They can't get anything 699.84s - 704.08s | via web inspection or TCP dump or any of that type of stuff to try to recover the firmware 704.08s - 712.24s | from network traffic. Oh, I'll go to the website. Nope, nothing on their website, you go to their 712.24s - 718.00s | support forums, you go to their support pages. Absolutely no firmware for the x 1800 available 718.00s - 724.72s | for download, there's manuals and stuff. But no firmware. So what happens if I break it by 724.72s - 728.80s | accident, I'd have no way to upload firmware to this because the manufacturer doesn't even make 728.80s - 733.76s | it available. I checked the support forums to see if I could find links somewhere in the support 733.76s - 740.64s | forums. And that was a mess. It's dabbling is a Korean company, they have two different support 740.64s - 747.84s | forums. One is in Korean, one is in English. The English has a little bit more population to it. 748.48s - 753.44s | In that more folks have asked some questions. The Korean version of the support forums, 754.16s - 759.68s | there's like three posts, and that's it. Clearly, Korea was not their intended market for these 759.68s - 765.60s | devices. And Google Translate had a really easy time trying to go through and translate that stuff 765.60s - 771.60s | because there wasn't a wasn't a lot there. search through all of the support forum posts, there are 771.60s - 776.08s | no links to firmware, there's nobody even asking, Hey, I bricked my router, can I get a copy of the 776.08s - 781.36s | firmware? So that I could reach out to them and try to ask them for it or try that same guys. 782.64s - 788.72s | And no, so nobody was asking and nobody was offering. Unfortunately, now that means with 788.72s - 795.20s | no ready access to the firmware, we've got to go do this the hard way. And no access ready access 795.20s - 801.84s | to firmware. That means no wrong way. That means no, no S bomb, no enrichment, no ability for me 801.84s - 806.08s | to figure out


chunk_text:
In that more folks have asked some questions. The Korean version of the support forums, 754.16s - 759.68s | there's like three posts, and that's it. Clearly, Korea was not their intended market for these 759.68s - 765.60s | devices. And Google Translate had a really easy time trying to go through and translate that stuff 765.60s - 771.60s | because there wasn't a wasn't a lot there. search through all of the support forum posts, there are 771.60s - 776.08s | no links to firmware, there's nobody even asking, Hey, I bricked my router, can I get a copy of the 776.08s - 781.36s | firmware? So that I could reach out to them and try to ask them for it or try that same guys. 782.64s - 788.72s | And no, so nobody was asking and nobody was offering. Unfortunately, now that means with 788.72s - 795.20s | no ready access to the firmware, we've got to go do this the hard way. And no access ready access 795.20s - 801.84s | to firmware. That means no wrong way. That means no, no S bomb, no enrichment, no ability for me 801.84s - 806.08s | to figure out if there are vulnerabilities in this really easily, based on software components, 806.08s - 814.72s | and so forth. So but no s bombs means no fun. Right? I love some of this stuff. I really want 814.72s - 822.80s | I tried to make this image like the dystopian future dark gray. And the training for the AI 822.80s - 827.36s | models at mid journey was not picking up that I wanted this to be dirty and gritty, no matter how 827.36s - 833.36s | many times I told it and it came out really happy and bright and green and very cartoony for all 833.36s - 840.72s | these dogs. No, I wanted like, like wolves that were gonna tear Bob's head off. And now there's 840.72s - 848.08s | some nice puppies here. So I did the best I could. All right. But so if we can't get ready access to 848.08s - 854.00s | it, we got to go about this do this hard way. And while brain surgery is fun, right? Yeah, just ask 854.72s - 858.32s | buckaroo bonsai adventures of buckaroo bonsai in the eighth dimension. 858.88s - 864.72s | Hey, this


chunk_text:
if there are vulnerabilities in this really easily, based on software components, 806.08s - 814.72s | and so forth. So but no s bombs means no fun. Right? I love some of this stuff. I really want 814.72s - 822.80s | I tried to make this image like the dystopian future dark gray. And the training for the AI 822.80s - 827.36s | models at mid journey was not picking up that I wanted this to be dirty and gritty, no matter how 827.36s - 833.36s | many times I told it and it came out really happy and bright and green and very cartoony for all 833.36s - 840.72s | these dogs. No, I wanted like, like wolves that were gonna tear Bob's head off. And now there's 840.72s - 848.08s | some nice puppies here. So I did the best I could. All right. But so if we can't get ready access to 848.08s - 854.00s | it, we got to go about this do this hard way. And while brain surgery is fun, right? Yeah, just ask 854.72s - 858.32s | buckaroo bonsai adventures of buckaroo bonsai in the eighth dimension. 858.88s - 864.72s | Hey, this was how some of this AI prompt ended up. I told mid journey that I wanted Bob the 864.72s - 870.64s | minion to go undergo brain surgery by buckaroo bonsai. Have the surgeon be buckaroo bonsai. 871.20s - 878.24s | Unfortunately, it actually crossed Peter Weller with a minion and came up with this abomination. 878.88s - 885.84s | It's quite amazing. Like this is the best of them. The other ones were just downright terrible. 887.04s - 892.32s | Also, if you've never seen buckaroo, the adventures of buckaroo bonsai in the eighth dimension, 892.32s - 898.00s | I highly recommend that you do so it is a 1980s very campy film with some really amazing stars. 899.04s - 905.28s | Peter Weller, John Lithgow, a whole host of folks and it's it's amazingly campy and maybe one for 905.28s - 911.84s | Jeff Moss to send a hacker movie night because it's it's hilarious and it's great. It's on the 911.84s - 919.12s | what the top 1000 movies you should see before you die and like number 43 in some of the campy 919.12s - 927.36s | sci fi movies, top lists and so forth. So


chunk_text:
was how some of this AI prompt ended up. I told mid journey that I wanted Bob the 864.72s - 870.64s | minion to go undergo brain surgery by buckaroo bonsai. Have the surgeon be buckaroo bonsai. 871.20s - 878.24s | Unfortunately, it actually crossed Peter Weller with a minion and came up with this abomination. 878.88s - 885.84s | It's quite amazing. Like this is the best of them. The other ones were just downright terrible. 887.04s - 892.32s | Also, if you've never seen buckaroo, the adventures of buckaroo bonsai in the eighth dimension, 892.32s - 898.00s | I highly recommend that you do so it is a 1980s very campy film with some really amazing stars. 899.04s - 905.28s | Peter Weller, John Lithgow, a whole host of folks and it's it's amazingly campy and maybe one for 905.28s - 911.84s | Jeff Moss to send a hacker movie night because it's it's hilarious and it's great. It's on the 911.84s - 919.12s | what the top 1000 movies you should see before you die and like number 43 in some of the campy 919.12s - 927.36s | sci fi movies, top lists and so forth. So my plug and note that Bob doesn't even have yellow ears 927.36s - 931.76s | in this one. They're apparently Peter Weller's ears. Not only did they do a brain transplant, 931.76s - 935.92s | they didn't hear transplant to. Alright, so let's get to this brain surgery. 937.28s - 942.56s | Alright, take this thing apart. We do this the hard way. Pull the boards out, do an inventory, 942.56s - 949.12s | take pictures of all the chips. Let's do some some analysis and figure out some of the spec 949.12s - 953.44s | sheets, go download those and then start figuring out how we can get access to that firmware. 954.32s - 958.56s | Are there serial ports available on the board? Yes, yes. In this case, there is I did not go 958.56s - 965.84s | that route. However, can we use JTAG or SWD to dump firmware from from the device from that bus? 966.64s - 970.72s | I fortunately did not have to go down that road either, because that is arguably the harder. 971.92s - 979.60s | But what about SPI? And being able to grab that from there? The first question was


chunk_text:
my plug and note that Bob doesn't even have yellow ears 927.36s - 931.76s | in this one. They're apparently Peter Weller's ears. Not only did they do a brain transplant, 931.76s - 935.92s | they didn't hear transplant to. Alright, so let's get to this brain surgery. 937.28s - 942.56s | Alright, take this thing apart. We do this the hard way. Pull the boards out, do an inventory, 942.56s - 949.12s | take pictures of all the chips. Let's do some some analysis and figure out some of the spec 949.12s - 953.44s | sheets, go download those and then start figuring out how we can get access to that firmware. 954.32s - 958.56s | Are there serial ports available on the board? Yes, yes. In this case, there is I did not go 958.56s - 965.84s | that route. However, can we use JTAG or SWD to dump firmware from from the device from that bus? 966.64s - 970.72s | I fortunately did not have to go down that road either, because that is arguably the harder. 971.92s - 979.60s | But what about SPI? And being able to grab that from there? The first question was looking at 979.60s - 987.04s | the processor. It's an RTL 8198 D featured here in this slide. And I figured I was probably going 987.04s - 992.16s | to have to go use JTAG have the processor with internal memory and have to dump it from internal 992.16s - 1003.84s | memory and so forth, but not so much. But what I did find on the board was a Winbond W25N01GV, 1004.64s - 1016.88s | which is a three volt one gigabit flash chip. But they said 128 Meg. No one gigabit available on 1016.88s - 1021.60s | this board. And they're only using 128 megabit according to what they're saying. Yeah, so there's 1021.60s - 1028.48s | a lot more room on this. Yeah, why? I don't know. Because they're cheap. Honestly. Ask me how I know 1028.48s - 1036.64s | this. We'll see this in a little bit here. So SPI, and it is NAND. So alright, this will be 1036.64s - 1043.68s | interesting. And apparently Bob is really hungry for some chips. Alright, so I had to go now get 1043.68s - 1050.64s | this flash off of this Winbond SPI flash chip. And there


chunk_text:
looking at 979.60s - 987.04s | the processor. It's an RTL 8198 D featured here in this slide. And I figured I was probably going 987.04s - 992.16s | to have to go use JTAG have the processor with internal memory and have to dump it from internal 992.16s - 1003.84s | memory and so forth, but not so much. But what I did find on the board was a Winbond W25N01GV, 1004.64s - 1016.88s | which is a three volt one gigabit flash chip. But they said 128 Meg. No one gigabit available on 1016.88s - 1021.60s | this board. And they're only using 128 megabit according to what they're saying. Yeah, so there's 1021.60s - 1028.48s | a lot more room on this. Yeah, why? I don't know. Because they're cheap. Honestly. Ask me how I know 1028.48s - 1036.64s | this. We'll see this in a little bit here. So SPI, and it is NAND. So alright, this will be 1036.64s - 1043.68s | interesting. And apparently Bob is really hungry for some chips. Alright, so I had to go now get 1043.68s - 1050.64s | this flash off of this Winbond SPI flash chip. And there was very much of progression of frustration 1050.64s - 1057.20s | going through going through this. So this is what society thinks I do when I remove firmware from a 1057.20s - 1065.44s | chip. Credit to Wired and IBM here, sorry, Intel, in some of their photographs about doing some 1065.44s - 1069.44s | really great work with a microscope and all sorts of fun stuff. And you'll note, I'm sort of 1069.44s - 1074.24s | featured down here in the right hand corner, looking very studious and doing a really great 1074.24s - 1081.84s | job of removing firmware and what society thinks I do. This is what my friends think I do using the 1081.84s - 1087.84s | PC bytes. And I'm starting to look a lot worse and maybe a little bit nerdier and geekier. And 1087.84s - 1093.04s | yes, this is my disaster of a desk and home office. And you get to see also other other fun 1093.04s - 1097.76s | stuff in the background. So this is what my friends think I do. Right, Mick? Yeah. All right. 1098.72s - 1104.48s | This is what I think I do. This is also


chunk_text:
was very much of progression of frustration 1050.64s - 1057.20s | going through going through this. So this is what society thinks I do when I remove firmware from a 1057.20s - 1065.44s | chip. Credit to Wired and IBM here, sorry, Intel, in some of their photographs about doing some 1065.44s - 1069.44s | really great work with a microscope and all sorts of fun stuff. And you'll note, I'm sort of 1069.44s - 1074.24s | featured down here in the right hand corner, looking very studious and doing a really great 1074.24s - 1081.84s | job of removing firmware and what society thinks I do. This is what my friends think I do using the 1081.84s - 1087.84s | PC bytes. And I'm starting to look a lot worse and maybe a little bit nerdier and geekier. And 1087.84s - 1093.04s | yes, this is my disaster of a desk and home office. And you get to see also other other fun 1093.04s - 1097.76s | stuff in the background. So this is what my friends think I do. Right, Mick? Yeah. All right. 1098.72s - 1104.48s | This is what I think I do. This is also one of my attempts, soldering wires directly to the 1104.56s - 1113.20s | flash chip and then holding it down with some tape. This didn't work either. Okay. So I'm 1113.20s - 1119.36s | getting worse and worse in my my studiousness here. And ultimately, this is what I really did. 1120.64s - 1125.60s | I now look like a really big goober. And I actually had to remove the chip from the board 1125.60s - 1132.72s | and put it in like a $3 adapter from Amazon and use some some DuPont wires and attach it to a 1132.72s - 1140.56s | Raspberry Pi and use that IO on the board for the Raspberry Pi. Yeah, and just reuse the PC bytes 1140.56s - 1145.28s | thing to hold my Raspberry Pi because I was too lazy to take it off my desk. So yeah, I actually 1145.28s - 1149.60s | had to do take the chip off of the board because I couldn't get any of the other stuff to work 1149.60s - 1154.24s | right. And it may not have been the fact that there was anything wrong with my PC bytes or 1154.24s - 1162.00s |


chunk_text:
one of my attempts, soldering wires directly to the 1104.56s - 1113.20s | flash chip and then holding it down with some tape. This didn't work either. Okay. So I'm 1113.20s - 1119.36s | getting worse and worse in my my studiousness here. And ultimately, this is what I really did. 1120.64s - 1125.60s | I now look like a really big goober. And I actually had to remove the chip from the board 1125.60s - 1132.72s | and put it in like a $3 adapter from Amazon and use some some DuPont wires and attach it to a 1132.72s - 1140.56s | Raspberry Pi and use that IO on the board for the Raspberry Pi. Yeah, and just reuse the PC bytes 1140.56s - 1145.28s | thing to hold my Raspberry Pi because I was too lazy to take it off my desk. So yeah, I actually 1145.28s - 1149.60s | had to do take the chip off of the board because I couldn't get any of the other stuff to work 1149.60s - 1154.24s | right. And it may not have been the fact that there was anything wrong with my PC bytes or 1154.24s - 1162.00s | with my solder job. But I do know that I bricked the one that I did the solder job to no longer 1162.00s - 1169.12s | functions. And we'll see why that maybe it wasn't actually the PC bytes or my solder job that 1169.12s - 1177.60s | wasn't allowing me to get access to the stuff. I see I thought I had an hour. Okay, I will I'll 1177.60s - 1182.00s | get out of your head here. What's a I'm going to do these slides real quick here what I really did. 1183.52s - 1188.56s | Not a lot of flash ROM or support found some stuff. I got a dump found some treasure, 1188.56s - 1193.92s | lots of exploits, lots of exploit intelligence, lots of remote code vulnerabilities, 1195.52s - 1201.36s | passwords, username, Bob password, Bob, other stuff in Korean language, some key material, 1202.40s - 1207.76s | a horrible web app that they wrote their own web server with some possible command injection via 1207.76s - 1215.04s | the password reset function, a bunch of scripts going to IP in China, all sorts of other fun 1215.04s - 1219.76s | stuff. And I really


chunk_text:
with my solder job. But I do know that I bricked the one that I did the solder job to no longer 1162.00s - 1169.12s | functions. And we'll see why that maybe it wasn't actually the PC bytes or my solder job that 1169.12s - 1177.60s | wasn't allowing me to get access to the stuff. I see I thought I had an hour. Okay, I will I'll 1177.60s - 1182.00s | get out of your head here. What's a I'm going to do these slides real quick here what I really did. 1183.52s - 1188.56s | Not a lot of flash ROM or support found some stuff. I got a dump found some treasure, 1188.56s - 1193.92s | lots of exploits, lots of exploit intelligence, lots of remote code vulnerabilities, 1195.52s - 1201.36s | passwords, username, Bob password, Bob, other stuff in Korean language, some key material, 1202.40s - 1207.76s | a horrible web app that they wrote their own web server with some possible command injection via 1207.76s - 1215.04s | the password reset function, a bunch of scripts going to IP in China, all sorts of other fun 1215.04s - 1219.76s | stuff. And I really want to do the brain transplant to fix the one that I broke. And I've 1219.76s - 1224.72s | had no luck. I spent a week on the struggle bus with that, because the Nan support that I needed 1224.72s - 1232.08s | has no right support. So coming soon. All right. I'm out of here. I'm gonna if you want to catch 1232.08s - 1236.64s | me in the back or over in the IT village, I'm happy to do some more. But I want to say thank 1236.64s - 1241.92s | you and a big thank you to my co worker Edwin shuttleworth, who is all now so now a minion as 1241.92s - 1246.88s | well. So thank you very much. And apologies for taking an extra moment of your time. And I'll see 1246.88s - 1249.92s | you over at the IoT village. We'll be right up front here. If you've got any questions, 1249.92s - 1253.44s | and you want to see some more of these slides and some of the URLs.


chunk_text:
want to do the brain transplant to fix the one that I broke. And I've 1219.76s - 1224.72s | had no luck. I spent a week on the struggle bus with that, because the Nan support that I needed 1224.72s - 1232.08s | has no right support. So coming soon. All right. I'm out of here. I'm gonna if you want to catch 1232.08s - 1236.64s | me in the back or over in the IT village, I'm happy to do some more. But I want to say thank 1236.64s - 1241.92s | you and a big thank you to my co worker Edwin shuttleworth, who is all now so now a minion as 1241.92s - 1246.88s | well. So thank you very much. And apologies for taking an extra moment of your time. And I'll see 1246.88s - 1249.92s | you over at the IoT village. We'll be right up front here. If you've got any questions, 1249.92s - 1253.44s | and you want to see some more of these slides and some of the URLs.


chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=PZfM1ehcfmU", "title": "DEF CON 32 - Pick Your Poison: Navigating a secure clean energy transition - Emma Stewart", "description": "Clean Energy technology, while essential for the energy transition, often utilizes components sourced from adversarial countries, potentially increasing cyber vulnerabilities, especially in systems managed by smaller utilities with limited cybersecurity expertise. The previous approaches of 'rip and replace' to eliminate foreign components have proved inadequate and if applied to clean energy, may be economically and technically destabilizing.\n\nThis policy session will also delve into the broader implications of a digitalized supply chain, where simplistic views of cybersecurity do not suffice. Instead, a nuanced understanding of the operational and physical realities of energy systems is essential. By discussing how to balance immediate economic pressures against long-term sustainability and security, the session aims to foster a more informed and effective policy discourse.\n\nIn summary, as the energy sector evolves to include diverse infrastructure systems\u2014ranging from hospitals to military installations\u2014the need for a robust, informed, and agile policy approach to cybersecurity becomes increasingly critical. This session will provide a platform for critical discourse to ensure that the transition to renewable energy is both secure and sustainable, aligning climate goals with national security imperatives", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1845, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 2.24s | This text was transcribed using whisper model: large-v2 Hi everybody, thanks for joining me today. 2.24s - 4.12s | Oh, that was loud, I'm sorry. 4.12s - 5.52s | Thanks for joining me today. 6.52s - 7.44s | My name's Emma Stewart. 7.44s - 9.28s | I'm a Chief Power Grid Scientist 9.28s - 11.48s | at Idaho National Laboratory. 11.48s - 13.36s | I've spent about the last 20 years 13.36s - 16.76s | working on power grid narrowed stuff effectively. 16.76s - 19.44s | I've done everything from essentially 19.44s - 23.32s | working on poles and wires about 20 something years ago 23.32s - 26.20s | up to now where I like to be called Doctor 26.20s - 29.84s | and it's kind of cool, but I do all sorts of things. 29.84s - 33.00s | I've worked as a Chief Scientist for 900 utilities. 33.00s - 37.64s | I built a cyber program for all of the small 37.64s - 40.40s | rural co-ops in the country. 40.40s - 42.40s | But the last few years I've been working 42.40s - 43.76s


chunk_text:
"https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1845, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 2.24s | This text was transcribed using whisper model: large-v2 Hi everybody, thanks for joining me today. 2.24s - 4.12s | Oh, that was loud, I'm sorry. 4.12s - 5.52s | Thanks for joining me today. 6.52s - 7.44s | My name's Emma Stewart. 7.44s - 9.28s | I'm a Chief Power Grid Scientist 9.28s - 11.48s | at Idaho National Laboratory. 11.48s - 13.36s | I've spent about the last 20 years 13.36s - 16.76s | working on power grid narrowed stuff effectively. 16.76s - 19.44s | I've done everything from essentially 19.44s - 23.32s | working on poles and wires about 20 something years ago 23.32s - 26.20s | up to now where I like to be called Doctor 26.20s - 29.84s | and it's kind of cool, but I do all sorts of things. 29.84s - 33.00s | I've worked as a Chief Scientist for 900 utilities. 33.00s - 37.64s | I built a cyber program for all of the small 37.64s - 40.40s | rural co-ops in the country. 40.40s - 42.40s | But the last few years I've been working 42.40s - 43.76s | mostly on renewable security 43.76s - 45.68s | and that's what I'm gonna talk about today. 45.68s - 48.80s | So just very briefly, I'll start here. 48.80s - 52.04s | We are in the middle of various different crises. 52.04s - 54.72s | Essentially the climate has been changing 54.72s - 56.76s | and in the last few years we've had, 56.76s - 59.72s | I believe a number of now $10 billion weather 60.60s - 61.64s | events that are causing all sorts of havoc 61.64s - 64.56s | around the country, especially in our power grid 64.56s - 66.76s | in particular because things fall down in weather. 66.76s - 68.52s | As much as everyone wants to think cyber 68.52s - 69.96s | is probably the worst problem we have, 69.96s - 71.32s | it's not, it's weather. 71.32s - 74.60s | It knocks things down far more often than any of you do. 74.60s - 78.96s | So yeah, that's the world. 78.96s - 80.60s | But this isn't great. 80.60s - 82.08s | We do need to change this. 82.08s - 84.28s | The energy sector essentially provides 84.28s - 88.20s | about


chunk_text:
| mostly on renewable security 43.76s - 45.68s | and that's what I'm gonna talk about today. 45.68s - 48.80s | So just very briefly, I'll start here. 48.80s - 52.04s | We are in the middle of various different crises. 52.04s - 54.72s | Essentially the climate has been changing 54.72s - 56.76s | and in the last few years we've had, 56.76s - 59.72s | I believe a number of now $10 billion weather 60.60s - 61.64s | events that are causing all sorts of havoc 61.64s - 64.56s | around the country, especially in our power grid 64.56s - 66.76s | in particular because things fall down in weather. 66.76s - 68.52s | As much as everyone wants to think cyber 68.52s - 69.96s | is probably the worst problem we have, 69.96s - 71.32s | it's not, it's weather. 71.32s - 74.60s | It knocks things down far more often than any of you do. 74.60s - 78.96s | So yeah, that's the world. 78.96s - 80.60s | But this isn't great. 80.60s - 82.08s | We do need to change this. 82.08s - 84.28s | The energy sector essentially provides 84.28s - 88.20s | about 28% of emissions into our atmosphere. 88.24s - 90.04s | That's helping with this climate change problem. 90.04s - 91.88s | So we're trying to change that by installing 91.88s - 93.12s | lots and lots of renewable energy 93.12s - 95.24s | and transitioning to this more clean, 95.24s - 97.92s | digital, affordable system also. 97.92s - 100.28s | But just, I am a power grid person, 100.28s - 101.84s | so in all of these talks I give a 101.84s - 103.44s | what is the power grid to everyone. 103.44s - 105.20s | If you come to one of my power grid talks 105.20s - 108.04s | at a power conference, you won't hear this part. 108.04s - 111.08s | But how does the system work and how has it worked? 111.08s - 112.88s | We've essentially for a very long time 112.88s - 115.56s | had power that flows from large spinning generators 115.56s - 117.76s | around the country all the way down 118.24s - 121.76s | through our large power lines down to small customers, 121.76s - 124.72s | sub-transmission, primary customers, your houses. 124.72s - 126.92s | That's been the way it's


chunk_text:
28% of emissions into our atmosphere. 88.24s - 90.04s | That's helping with this climate change problem. 90.04s - 91.88s | So we're trying to change that by installing 91.88s - 93.12s | lots and lots of renewable energy 93.12s - 95.24s | and transitioning to this more clean, 95.24s - 97.92s | digital, affordable system also. 97.92s - 100.28s | But just, I am a power grid person, 100.28s - 101.84s | so in all of these talks I give a 101.84s - 103.44s | what is the power grid to everyone. 103.44s - 105.20s | If you come to one of my power grid talks 105.20s - 108.04s | at a power conference, you won't hear this part. 108.04s - 111.08s | But how does the system work and how has it worked? 111.08s - 112.88s | We've essentially for a very long time 112.88s - 115.56s | had power that flows from large spinning generators 115.56s - 117.76s | around the country all the way down 118.24s - 121.76s | through our large power lines down to small customers, 121.76s - 124.72s | sub-transmission, primary customers, your houses. 124.72s - 126.92s | That's been the way it's been for a very long time. 126.92s - 129.16s | Since the beginning of all power grid time, 129.16s - 130.76s | it has looked this way. 130.76s - 131.60s | And that's changing. 131.60s - 133.80s | We have massive amounts of things happening 133.80s - 136.04s | to do with energy storage, which is the primary focus 136.04s - 138.08s | of this discussion today. 138.08s - 140.64s | We have electric vehicles plugging in everywhere. 140.64s - 142.64s | We also have things like the cloud. 142.64s - 144.20s | A lot of the small utilities are trying 144.20s - 146.64s | to modernize at the moment and their only option 146.68s - 149.72s | for modernization is actually using cloud-based applications 149.72s - 151.08s | in lots of places. 151.08s - 154.16s | They can't install data centers in tiny utilities. 154.16s - 156.20s | I will say I've seen the convergence 156.20s - 157.20s | of these things recently. 157.20s - 158.72s | I was working with a small utility 158.72s - 162.48s | and they had had a massive tornado come through 162.48s - 164.44s | that took out one of their operations center


chunk_text:
been for a very long time. 126.92s - 129.16s | Since the beginning of all power grid time, 129.16s - 130.76s | it has looked this way. 130.76s - 131.60s | And that's changing. 131.60s - 133.80s | We have massive amounts of things happening 133.80s - 136.04s | to do with energy storage, which is the primary focus 136.04s - 138.08s | of this discussion today. 138.08s - 140.64s | We have electric vehicles plugging in everywhere. 140.64s - 142.64s | We also have things like the cloud. 142.64s - 144.20s | A lot of the small utilities are trying 144.20s - 146.64s | to modernize at the moment and their only option 146.68s - 149.72s | for modernization is actually using cloud-based applications 149.72s - 151.08s | in lots of places. 151.08s - 154.16s | They can't install data centers in tiny utilities. 154.16s - 156.20s | I will say I've seen the convergence 156.20s - 157.20s | of these things recently. 157.20s - 158.72s | I was working with a small utility 158.72s - 162.48s | and they had had a massive tornado come through 162.48s - 164.44s | that took out one of their operations center 164.44s - 168.44s | and their data center was effectively Dorothied into a river. 168.44s - 172.08s | So there's a lot happening together in the power grid 172.08s - 174.60s | and things there are there, so. 174.60s - 176.44s | But what does energy transition mean to me 176.44s - 178.60s | and to a lot of people? 178.60s - 179.96s | I said big spinning machines 179.96s - 183.00s | as in these big rotating generators that are there. 183.00s - 184.56s | They've been there forever. 184.56s - 185.88s | They're very reliable. 185.88s - 187.44s | They've been around a long time. 187.44s - 189.84s | Big spinning metal, effectively. 189.84s - 191.92s | They've been traditionally manufactured around the country 191.92s - 194.72s | and most of that has a decent sized U.S. base 194.72s - 196.24s | for manufacture. 196.24s - 197.92s | I'll use the spoiler alert here. 197.92s - 200.32s | As we transition to things like solar storage, 200.32s - 202.68s | lots of distributed systems, 202.72s - 204.60s | it's gonna be massive distributed connected 204.60s - 206.76s | solid state devices all over the place 206.76s - 208.20s


chunk_text:
164.44s - 168.44s | and their data center was effectively Dorothied into a river. 168.44s - 172.08s | So there's a lot happening together in the power grid 172.08s - 174.60s | and things there are there, so. 174.60s - 176.44s | But what does energy transition mean to me 176.44s - 178.60s | and to a lot of people? 178.60s - 179.96s | I said big spinning machines 179.96s - 183.00s | as in these big rotating generators that are there. 183.00s - 184.56s | They've been there forever. 184.56s - 185.88s | They're very reliable. 185.88s - 187.44s | They've been around a long time. 187.44s - 189.84s | Big spinning metal, effectively. 189.84s - 191.92s | They've been traditionally manufactured around the country 191.92s - 194.72s | and most of that has a decent sized U.S. base 194.72s - 196.24s | for manufacture. 196.24s - 197.92s | I'll use the spoiler alert here. 197.92s - 200.32s | As we transition to things like solar storage, 200.32s - 202.68s | lots of distributed systems, 202.72s - 204.60s | it's gonna be massive distributed connected 204.60s - 206.76s | solid state devices all over the place 206.76s - 208.20s | with lots of numbers of them 208.20s - 209.76s | where there was one big generator 209.76s - 212.12s | that now might be 10,000 smaller devices 212.12s - 215.24s | connected together, providing power to our grid. 215.24s - 218.04s | And the spoiler alert is with largely Chinese manufacturers. 218.04s - 220.76s | So most of this is manufactured in China 220.76s - 222.08s | and that's why I'm talking today. 222.08s - 224.32s | But if you like the light's been on today, 224.32s - 227.36s | this is just what this means to you in the room. 227.36s - 229.60s | The MGM is the biggest solar rooftop 229.60s - 231.24s | in the world at the moment. 231.24s - 233.52s | If you can see on top of the MGM, 233.52s - 235.72s | that's apparently the largest owner of solar 235.72s - 237.36s | in this area on a rooftop. 237.36s - 239.00s | It's kind of amazing actually 239.00s - 240.64s | when you look at the rooftops of Vegas, 240.64s - 242.92s | you start to see this solar stuff everywhere. 243.88s - 246.32s | The energy storage is actually wildly


chunk_text:
| with lots of numbers of them 208.20s - 209.76s | where there was one big generator 209.76s - 212.12s | that now might be 10,000 smaller devices 212.12s - 215.24s | connected together, providing power to our grid. 215.24s - 218.04s | And the spoiler alert is with largely Chinese manufacturers. 218.04s - 220.76s | So most of this is manufactured in China 220.76s - 222.08s | and that's why I'm talking today. 222.08s - 224.32s | But if you like the light's been on today, 224.32s - 227.36s | this is just what this means to you in the room. 227.36s - 229.60s | The MGM is the biggest solar rooftop 229.60s - 231.24s | in the world at the moment. 231.24s - 233.52s | If you can see on top of the MGM, 233.52s - 235.72s | that's apparently the largest owner of solar 235.72s - 237.36s | in this area on a rooftop. 237.36s - 239.00s | It's kind of amazing actually 239.00s - 240.64s | when you look at the rooftops of Vegas, 240.64s - 242.92s | you start to see this solar stuff everywhere. 243.88s - 246.32s | The energy storage is actually wildly useful 246.32s - 247.68s | just now for us. 247.68s - 250.88s | Both Texas and California have had major events 250.88s - 253.32s | that required them to ramp up fast generators 253.32s - 255.08s | to provide power really quickly. 256.12s - 258.40s | Both of them, I think it's been around 17 emergency 258.44s - 262.48s | dispatches of storage generation in California, 262.48s - 264.52s | I believe in the last couple of years, 264.52s - 266.68s | meaning a grid operator sitting there going, 266.68s - 268.64s | we need power really fast, 268.64s - 270.80s | asking them to do an immediate control action 270.80s - 272.96s | to discharge all of their power. 272.96s - 274.08s | That's kind of a big deal. 274.08s - 275.68s | It's an immediate action that needs to be taken 275.68s - 277.00s | to stop the lights going off. 277.00s - 279.56s | So this energy storage has been doing it. 279.56s - 281.68s | That picture on the right was in Texas, 281.68s - 285.68s | I believe January this year when a gigawatt of batteries 285.68s - 287.96s | ramped up to sort of


chunk_text:
useful 246.32s - 247.68s | just now for us. 247.68s - 250.88s | Both Texas and California have had major events 250.88s - 253.32s | that required them to ramp up fast generators 253.32s - 255.08s | to provide power really quickly. 256.12s - 258.40s | Both of them, I think it's been around 17 emergency 258.44s - 262.48s | dispatches of storage generation in California, 262.48s - 264.52s | I believe in the last couple of years, 264.52s - 266.68s | meaning a grid operator sitting there going, 266.68s - 268.64s | we need power really fast, 268.64s - 270.80s | asking them to do an immediate control action 270.80s - 272.96s | to discharge all of their power. 272.96s - 274.08s | That's kind of a big deal. 274.08s - 275.68s | It's an immediate action that needs to be taken 275.68s - 277.00s | to stop the lights going off. 277.00s - 279.56s | So this energy storage has been doing it. 279.56s - 281.68s | That picture on the right was in Texas, 281.68s - 285.68s | I believe January this year when a gigawatt of batteries 285.68s - 287.96s | ramped up to sort of save the day. 288.48s - 289.76s | And that was a good thing. 290.60s - 293.72s | But current state of supply, this is a weird graph, 293.72s - 296.84s | but last year after this conference, 296.84s - 299.32s | I was at another renewable energy conference 299.32s - 300.16s | in the same place. 300.16s - 301.88s | It's the biggest renewable energy conference, 301.88s - 303.12s | I believe, in the world. 304.24s - 306.16s | I've worked in renewable energy for a long time 306.16s - 309.56s | and sort of watched the change of the supply chain here. 309.56s - 312.00s | Me and one of my other cyber colleagues, 312.00s - 313.84s | there was only about three of us there, 313.84s - 316.20s | were walking around the shop floor, 316.20s - 317.48s | the expo floor as you do, 318.00s - 319.04s | trying to look at all the gear 319.04s - 321.16s | and realized that almost everything we were looking at 321.16s - 322.40s | was a Chinese company. 322.40s - 325.16s | It wasn't hidden, it wasn't covert, 325.16s - 327.40s | it wasn't a mysterious component we were


chunk_text:
save the day. 288.48s - 289.76s | And that was a good thing. 290.60s - 293.72s | But current state of supply, this is a weird graph, 293.72s - 296.84s | but last year after this conference, 296.84s - 299.32s | I was at another renewable energy conference 299.32s - 300.16s | in the same place. 300.16s - 301.88s | It's the biggest renewable energy conference, 301.88s - 303.12s | I believe, in the world. 304.24s - 306.16s | I've worked in renewable energy for a long time 306.16s - 309.56s | and sort of watched the change of the supply chain here. 309.56s - 312.00s | Me and one of my other cyber colleagues, 312.00s - 313.84s | there was only about three of us there, 313.84s - 316.20s | were walking around the shop floor, 316.20s - 317.48s | the expo floor as you do, 318.00s - 319.04s | trying to look at all the gear 319.04s - 321.16s | and realized that almost everything we were looking at 321.16s - 322.40s | was a Chinese company. 322.40s - 325.16s | It wasn't hidden, it wasn't covert, 325.16s - 327.40s | it wasn't a mysterious component we were finding 327.40s - 329.68s | in a device in critical infrastructure. 329.68s - 331.52s | It was straight up people that are, 331.52s - 334.60s | everyone is from China essentially was at this conference. 334.60s - 336.08s | And it was kind of a moment of change 336.08s - 337.80s | where before I would have said, 337.80s - 339.88s | we've got US manufacturer, we're building this. 339.88s - 341.08s | And I was kind of looking at it like, 341.08s - 344.64s | ah, I'm not sure we do and we're not hiding it. 344.64s - 346.24s | So this is a big challenge. 346.92s - 349.60s | I used this example here of, 349.60s - 351.84s | this was what the conference expo floor looked like. 351.84s - 354.16s | Orange is everything is from China 355.08s - 357.68s | and every other color is anywhere else. 357.68s - 359.60s | I think green I'd put in was Germany 359.60s - 362.60s | and blue is the US manufacturer as well. 362.60s - 366.24s | So about 95 to 100% of our digital components available 366.24s - 369.96s | right now have a Chinese made, owned


chunk_text:
finding 327.40s - 329.68s | in a device in critical infrastructure. 329.68s - 331.52s | It was straight up people that are, 331.52s - 334.60s | everyone is from China essentially was at this conference. 334.60s - 336.08s | And it was kind of a moment of change 336.08s - 337.80s | where before I would have said, 337.80s - 339.88s | we've got US manufacturer, we're building this. 339.88s - 341.08s | And I was kind of looking at it like, 341.08s - 344.64s | ah, I'm not sure we do and we're not hiding it. 344.64s - 346.24s | So this is a big challenge. 346.92s - 349.60s | I used this example here of, 349.60s - 351.84s | this was what the conference expo floor looked like. 351.84s - 354.16s | Orange is everything is from China 355.08s - 357.68s | and every other color is anywhere else. 357.68s - 359.60s | I think green I'd put in was Germany 359.60s - 362.60s | and blue is the US manufacturer as well. 362.60s - 366.24s | So about 95 to 100% of our digital components available 366.24s - 369.96s | right now have a Chinese made, owned or derived 369.96s - 372.32s | control or software component in it as well. 372.32s - 374.84s | So that is where we're at. 374.84s - 377.92s | But another sort of supply chain event that happened 377.92s - 381.24s | has been my life for the past two years actually. 381.24s - 386.24s | In April, 2023, a large military base 386.52s - 389.76s | announced they were installing and I think commissioning, 389.76s - 392.12s | the picture on the bottom left is that one, 392.12s - 395.96s | commissioning their giant new battery on Camp Lejeune. 395.96s - 398.20s | And basically they took a nice picture 398.20s - 400.12s | with the command cutting the ribbon 400.12s - 402.40s | with cattle written behind their head, 402.40s - 405.12s | which is one of the largest battery manufacturers 405.12s - 407.92s | in the world who is a Chinese company. 407.92s - 410.40s | And so given the defense posture that we look at 410.40s - 412.64s | from components that we're using, 412.64s - 414.80s | this was definitely a moment of us all looking at the TV 414.80s - 416.68s | going, why'd you


chunk_text:
or derived 369.96s - 372.32s | control or software component in it as well. 372.32s - 374.84s | So that is where we're at. 374.84s - 377.92s | But another sort of supply chain event that happened 377.92s - 381.24s | has been my life for the past two years actually. 381.24s - 386.24s | In April, 2023, a large military base 386.52s - 389.76s | announced they were installing and I think commissioning, 389.76s - 392.12s | the picture on the bottom left is that one, 392.12s - 395.96s | commissioning their giant new battery on Camp Lejeune. 395.96s - 398.20s | And basically they took a nice picture 398.20s - 400.12s | with the command cutting the ribbon 400.12s - 402.40s | with cattle written behind their head, 402.40s - 405.12s | which is one of the largest battery manufacturers 405.12s - 407.92s | in the world who is a Chinese company. 407.92s - 410.40s | And so given the defense posture that we look at 410.40s - 412.64s | from components that we're using, 412.64s - 414.80s | this was definitely a moment of us all looking at the TV 414.80s - 416.68s | going, why'd you take that selfie? 416.68s - 417.80s | Come on. 417.80s - 420.12s | So everyone sort of started to catch on 420.12s - 422.16s | that this was where we were at. 422.16s - 423.88s | Again, this led to a series of events 423.88s - 425.76s | that have been the rest of my life, 425.76s - 428.52s | which was dealing with there being various, 428.52s - 430.16s | like we're saying that Chinese hackers 430.16s - 432.44s | are preparing to attack our infrastructure. 432.44s - 433.28s | That is known. 434.20s - 437.52s | Following that, about a year later, this was in December, 437.52s - 439.24s | the large utility announced 439.24s - 440.92s | that they were disconnecting that battery 440.92s - 444.20s | that had been put on Camp Lejeune, 444.20s - 446.28s | citing there was security concerns with it. 446.28s - 447.52s | Okay. 447.52s - 451.40s | But this led to about 10 or 20 utilities in the country 451.40s - 456.28s | starting to call and say, wait, don't we have those as well? 456.28s - 457.32s | Are we meant to do something? 457.32s - 459.20s | And


chunk_text:
take that selfie? 416.68s - 417.80s | Come on. 417.80s - 420.12s | So everyone sort of started to catch on 420.12s - 422.16s | that this was where we were at. 422.16s - 423.88s | Again, this led to a series of events 423.88s - 425.76s | that have been the rest of my life, 425.76s - 428.52s | which was dealing with there being various, 428.52s - 430.16s | like we're saying that Chinese hackers 430.16s - 432.44s | are preparing to attack our infrastructure. 432.44s - 433.28s | That is known. 434.20s - 437.52s | Following that, about a year later, this was in December, 437.52s - 439.24s | the large utility announced 439.24s - 440.92s | that they were disconnecting that battery 440.92s - 444.20s | that had been put on Camp Lejeune, 444.20s - 446.28s | citing there was security concerns with it. 446.28s - 447.52s | Okay. 447.52s - 451.40s | But this led to about 10 or 20 utilities in the country 451.40s - 456.28s | starting to call and say, wait, don't we have those as well? 456.28s - 457.32s | Are we meant to do something? 457.32s - 459.20s | And we're like, ah, okay. 459.60s - 460.56s | I was like, first of all, 460.56s - 462.16s | can I get that battery for a minute? 462.16s - 463.56s | And then second, they were like, 463.56s - 465.48s | everyone was just panicking at this point 465.48s - 467.28s | that both their boards were yelling at them 467.28s - 470.00s | and they were concerned from a security perspective 470.00s - 473.48s | and asking for resources for how are they going to secure 473.48s - 476.12s | this because they can't get rid of it. 476.12s - 479.40s | They can't afford to ditch their $10 million investment 479.40s - 481.32s | and just buy something else. 481.32s - 483.68s | And even then there wasn't anything else for them to buy. 483.68s - 486.24s | So a lot of people were calling and panicking 486.24s - 488.32s | and various things going on. 488.36s - 491.24s | Then I will say we move on to now. 491.24s - 494.24s | I believe the largest solar plus storage project 494.24s - 498.40s | in the world came online about a couple months ago. 498.40s - 501.16s |


chunk_text:
we're like, ah, okay. 459.60s - 460.56s | I was like, first of all, 460.56s - 462.16s | can I get that battery for a minute? 462.16s - 463.56s | And then second, they were like, 463.56s - 465.48s | everyone was just panicking at this point 465.48s - 467.28s | that both their boards were yelling at them 467.28s - 470.00s | and they were concerned from a security perspective 470.00s - 473.48s | and asking for resources for how are they going to secure 473.48s - 476.12s | this because they can't get rid of it. 476.12s - 479.40s | They can't afford to ditch their $10 million investment 479.40s - 481.32s | and just buy something else. 481.32s - 483.68s | And even then there wasn't anything else for them to buy. 483.68s - 486.24s | So a lot of people were calling and panicking 486.24s - 488.32s | and various things going on. 488.36s - 491.24s | Then I will say we move on to now. 491.24s - 494.24s | I believe the largest solar plus storage project 494.24s - 498.40s | in the world came online about a couple months ago. 498.40s - 501.16s | It is connected to another military base also. 501.16s - 502.68s | When they announced this, they were saying, 502.68s - 504.56s | great, we've got our new batteries here. 504.56s - 506.96s | And then it was another one of the battery companies 506.96s - 509.72s | that's listed just now on the banned battery list 509.72s - 512.32s | on the National Defense Authorization Act. 512.32s - 514.08s | They announced it in the news though. 514.08s - 516.08s | So every time I read one of these news stories, 516.08s - 518.00s | I'm like, but there's a list 518.68s - 519.88s | and it's written and we're okay. 519.88s - 521.20s | And then they announced it on the news 521.20s - 522.84s | and more calls start. 522.84s - 524.00s | So that's been my life. 524.88s - 527.48s | The other thing that happened a few years ago, 527.48s - 529.40s | we obviously banned Huawei. 529.40s - 530.96s | That was a big deal. 530.96s - 531.96s | They actually had the market 531.96s - 534.04s | for various things of equipment. 534.04s - 537.24s | And about, I think four days ago,


chunk_text:
It is connected to another military base also. 501.16s - 502.68s | When they announced this, they were saying, 502.68s - 504.56s | great, we've got our new batteries here. 504.56s - 506.96s | And then it was another one of the battery companies 506.96s - 509.72s | that's listed just now on the banned battery list 509.72s - 512.32s | on the National Defense Authorization Act. 512.32s - 514.08s | They announced it in the news though. 514.08s - 516.08s | So every time I read one of these news stories, 516.08s - 518.00s | I'm like, but there's a list 518.68s - 519.88s | and it's written and we're okay. 519.88s - 521.20s | And then they announced it on the news 521.20s - 522.84s | and more calls start. 522.84s - 524.00s | So that's been my life. 524.88s - 527.48s | The other thing that happened a few years ago, 527.48s - 529.40s | we obviously banned Huawei. 529.40s - 530.96s | That was a big deal. 530.96s - 531.96s | They actually had the market 531.96s - 534.04s | for various things of equipment. 534.04s - 537.24s | And about, I think four days ago, 537.24s - 538.76s | Pentagon officials were announcing 538.76s - 542.20s | that they would like a waiver from the Huawei ban now 542.20s - 544.40s | because they can't get rid of the equipment. 544.40s - 547.12s | So if the Pentagon's asking for the waiver, 547.16s - 550.12s | rip and replace probably hasn't worked that well for Huawei. 550.12s - 552.68s | So going the same approach with the batteries 552.68s - 555.88s | is essentially the policy topic of my talk. 556.92s - 558.36s | So how'd we get here? 558.36s - 560.20s | Why did we end up this way? 560.20s - 561.64s | If you read any news stories, 561.64s - 565.36s | they try and blame whoever is in leadership right now. 565.36s - 567.40s | It's always the latest leader's problem 567.40s - 568.88s | that this has happened. 568.88s - 572.20s | But we started this probably 10 or 12 years ago. 572.20s - 574.16s | We prioritized low cost and volume 574.16s - 576.44s | for solar storage and renewable energy. 576.44s - 577.28s | It's great. 577.28s - 579.28s | We needed cheaper products for this to actually happen


chunk_text:
537.24s - 538.76s | Pentagon officials were announcing 538.76s - 542.20s | that they would like a waiver from the Huawei ban now 542.20s - 544.40s | because they can't get rid of the equipment. 544.40s - 547.12s | So if the Pentagon's asking for the waiver, 547.16s - 550.12s | rip and replace probably hasn't worked that well for Huawei. 550.12s - 552.68s | So going the same approach with the batteries 552.68s - 555.88s | is essentially the policy topic of my talk. 556.92s - 558.36s | So how'd we get here? 558.36s - 560.20s | Why did we end up this way? 560.20s - 561.64s | If you read any news stories, 561.64s - 565.36s | they try and blame whoever is in leadership right now. 565.36s - 567.40s | It's always the latest leader's problem 567.40s - 568.88s | that this has happened. 568.88s - 572.20s | But we started this probably 10 or 12 years ago. 572.20s - 574.16s | We prioritized low cost and volume 574.16s - 576.44s | for solar storage and renewable energy. 576.44s - 577.28s | It's great. 577.28s - 579.28s | We needed cheaper products for this to actually happen 579.28s - 580.92s | because we do need renewable energy. 580.92s - 584.56s | That's the sort of theme here is we need this to happen. 584.56s - 587.92s | But how we got there, we prioritized low cost. 587.92s - 589.84s | That meant we shift a lot of this offshore. 589.84s - 591.08s | That's just how it happens. 591.08s - 593.64s | We've started to manufacture it in China. 593.64s - 595.24s | And then we did get that low cost. 595.24s - 597.92s | We reached a dollar a watt in 2015, I believe. 597.92s - 600.20s | And then it's gone down from there. 600.20s - 601.04s | But now we're at this point, 601.04s - 604.28s | we just don't really build a lot of this in the US. 604.28s - 605.92s | We might soon. 605.92s - 608.80s | But investors need a demand signal. 608.80s - 610.88s | If we said we're gonna get rid of all the batteries 610.88s - 612.92s | and we're not gonna do this anymore, 612.92s - 616.04s | the investors aren't going to invest in new batteries 616.04s - 618.20s | being built here for another


chunk_text:
579.28s - 580.92s | because we do need renewable energy. 580.92s - 584.56s | That's the sort of theme here is we need this to happen. 584.56s - 587.92s | But how we got there, we prioritized low cost. 587.92s - 589.84s | That meant we shift a lot of this offshore. 589.84s - 591.08s | That's just how it happens. 591.08s - 593.64s | We've started to manufacture it in China. 593.64s - 595.24s | And then we did get that low cost. 595.24s - 597.92s | We reached a dollar a watt in 2015, I believe. 597.92s - 600.20s | And then it's gone down from there. 600.20s - 601.04s | But now we're at this point, 601.04s - 604.28s | we just don't really build a lot of this in the US. 604.28s - 605.92s | We might soon. 605.92s - 608.80s | But investors need a demand signal. 608.80s - 610.88s | If we said we're gonna get rid of all the batteries 610.88s - 612.92s | and we're not gonna do this anymore, 612.92s - 616.04s | the investors aren't going to invest in new batteries 616.04s - 618.20s | being built here for another five years. 618.20s - 620.44s | So what will we do in the interim? 620.44s - 622.00s | I use this picture on the right. 622.00s - 626.64s | It was from Bloomberg a few, about four years ago. 626.64s - 627.48s | It was actually, again, 627.48s - 629.96s | I love to use media stories as being funny. 629.96s - 631.92s | But it said basically, 631.92s - 635.36s | the US is number two in the world on battery manufacture. 635.88s - 638.12s | They are not that big line, so we're clear. 638.12s - 641.48s | We are not the large line, that was China. 641.48s - 642.60s | So we are number two, 642.60s - 645.52s | but we also lost by about eight laps of this. 645.52s - 647.12s | So we've got a lot of making up to do 647.12s - 649.56s | if we want to be competing in here. 649.56s - 652.24s | Just some quick facts and figures quickly. 652.24s - 654.64s | Huawei is still the number one inverter 654.64s - 658.36s | or power conversion manufacturer in the world. 658.36s - 660.60s | When they exited the market


chunk_text:
five years. 618.20s - 620.44s | So what will we do in the interim? 620.44s - 622.00s | I use this picture on the right. 622.00s - 626.64s | It was from Bloomberg a few, about four years ago. 626.64s - 627.48s | It was actually, again, 627.48s - 629.96s | I love to use media stories as being funny. 629.96s - 631.92s | But it said basically, 631.92s - 635.36s | the US is number two in the world on battery manufacture. 635.88s - 638.12s | They are not that big line, so we're clear. 638.12s - 641.48s | We are not the large line, that was China. 641.48s - 642.60s | So we are number two, 642.60s - 645.52s | but we also lost by about eight laps of this. 645.52s - 647.12s | So we've got a lot of making up to do 647.12s - 649.56s | if we want to be competing in here. 649.56s - 652.24s | Just some quick facts and figures quickly. 652.24s - 654.64s | Huawei is still the number one inverter 654.64s - 658.36s | or power conversion manufacturer in the world. 658.36s - 660.60s | When they exited the market from the US, 660.60s - 661.72s | they went from number two 661.72s - 664.20s | to number one in the world somehow. 664.20s - 666.16s | So that didn't necessarily work. 666.16s - 667.12s | What makes it different? 667.12s - 669.12s | Volume and regulation. 669.12s - 672.12s | There's about 73 named inverter manufacturers. 672.12s - 675.80s | So when I said we have large spinning generators, 675.80s - 677.44s | those are generally made by probably 677.44s - 679.88s | about three or four companies at this point. 679.88s - 681.84s | When we look at inverters and more particularly 681.84s - 684.40s | look at vulnerability analysis of inverters, 684.40s - 685.84s | you've got 73 companies 685.84s - 688.28s | and about 1,500 different models to look at 688.28s - 691.12s | that we haven't done that type of analysis on yet. 691.12s - 692.80s | And that's just now. 692.80s - 694.36s | If you look at battery vendors as well, 694.36s - 695.84s | we're in the same place. 695.84s - 697.88s | There's integrators of these equipment. 697.88s - 699.80s | A lot of the equipment is actually same hardware, 699.80s


chunk_text:
from the US, 660.60s - 661.72s | they went from number two 661.72s - 664.20s | to number one in the world somehow. 664.20s - 666.16s | So that didn't necessarily work. 666.16s - 667.12s | What makes it different? 667.12s - 669.12s | Volume and regulation. 669.12s - 672.12s | There's about 73 named inverter manufacturers. 672.12s - 675.80s | So when I said we have large spinning generators, 675.80s - 677.44s | those are generally made by probably 677.44s - 679.88s | about three or four companies at this point. 679.88s - 681.84s | When we look at inverters and more particularly 681.84s - 684.40s | look at vulnerability analysis of inverters, 684.40s - 685.84s | you've got 73 companies 685.84s - 688.28s | and about 1,500 different models to look at 688.28s - 691.12s | that we haven't done that type of analysis on yet. 691.12s - 692.80s | And that's just now. 692.80s - 694.36s | If you look at battery vendors as well, 694.36s - 695.84s | we're in the same place. 695.84s - 697.88s | There's integrators of these equipment. 697.88s - 699.80s | A lot of the equipment is actually same hardware, 699.80s - 701.24s | different label. 701.24s - 703.68s | Still, we need to do a lot of that analysis. 703.68s - 707.00s | But the hardware is often similar. 707.00s - 708.44s | And then we also have features 708.44s - 710.84s | like remote communication and access as necessary 710.84s - 713.24s | because if you have lots and lots and lots of devices, 713.24s - 715.48s | you kind of need them to get updated as well 715.48s - 717.56s | with all those nice patches that we need all the time. 717.56s - 719.52s | So remote comms is a big deal. 720.52s - 723.28s | There are some common challenges we have 723.28s - 725.92s | in renewable control systems in particular. 725.92s - 727.64s | They work pretty well just now. 727.64s - 729.56s | But again, inverters and power electronics, 729.56s - 733.20s | we see spinning generators last 40, 50 years. 733.20s - 735.16s | They're mostly older than all of us. 736.16s - 738.68s | Well, some of us, not looking at anyone else. 738.68s - 741.76s | But they're mostly older than all of us. 741.76s - 745.48s | But inverters


chunk_text:
- 701.24s | different label. 701.24s - 703.68s | Still, we need to do a lot of that analysis. 703.68s - 707.00s | But the hardware is often similar. 707.00s - 708.44s | And then we also have features 708.44s - 710.84s | like remote communication and access as necessary 710.84s - 713.24s | because if you have lots and lots and lots of devices, 713.24s - 715.48s | you kind of need them to get updated as well 715.48s - 717.56s | with all those nice patches that we need all the time. 717.56s - 719.52s | So remote comms is a big deal. 720.52s - 723.28s | There are some common challenges we have 723.28s - 725.92s | in renewable control systems in particular. 725.92s - 727.64s | They work pretty well just now. 727.64s - 729.56s | But again, inverters and power electronics, 729.56s - 733.20s | we see spinning generators last 40, 50 years. 733.20s - 735.16s | They're mostly older than all of us. 736.16s - 738.68s | Well, some of us, not looking at anyone else. 738.68s - 741.76s | But they're mostly older than all of us. 741.76s - 745.48s | But inverters tend to fail about every 15 years. 745.48s - 747.96s | So we actually have opportunity to get rid of them 747.96s - 749.56s | and get new ones eventually. 749.56s - 751.92s | But they do fail about every 15 years. 751.92s - 753.80s | Things that cause challenges with them. 753.80s - 754.68s | Failing chips. 754.68s - 756.76s | We have chipsets that just don't work very well 756.76s - 758.64s | in lots of cheap systems. 758.64s - 760.08s | Persistent communications. 760.08s - 762.08s | We have to communicate with these devices 762.08s - 762.92s | for them to work, 762.92s - 765.72s | but that's obviously a threat or a risk. 765.72s - 768.68s | Hard-coded passwords are freaking everywhere. 768.68s - 771.64s | Like, absolutely everywhere. 771.64s - 773.04s | And we can't, I'll talk why, 773.04s - 775.72s | but we can't do a ton about that right now. 775.76s - 778.68s | Operations and maintenance is a fourth-party person 778.68s - 780.04s | that will come in and do things. 780.04s - 781.96s | There's white-labeled products. 781.96s - 783.48s | Insecure applications. 783.48s - 786.20s | Some of the primary


chunk_text:
tend to fail about every 15 years. 745.48s - 747.96s | So we actually have opportunity to get rid of them 747.96s - 749.56s | and get new ones eventually. 749.56s - 751.92s | But they do fail about every 15 years. 751.92s - 753.80s | Things that cause challenges with them. 753.80s - 754.68s | Failing chips. 754.68s - 756.76s | We have chipsets that just don't work very well 756.76s - 758.64s | in lots of cheap systems. 758.64s - 760.08s | Persistent communications. 760.08s - 762.08s | We have to communicate with these devices 762.08s - 762.92s | for them to work, 762.92s - 765.72s | but that's obviously a threat or a risk. 765.72s - 768.68s | Hard-coded passwords are freaking everywhere. 768.68s - 771.64s | Like, absolutely everywhere. 771.64s - 773.04s | And we can't, I'll talk why, 773.04s - 775.72s | but we can't do a ton about that right now. 775.76s - 778.68s | Operations and maintenance is a fourth-party person 778.68s - 780.04s | that will come in and do things. 780.04s - 781.96s | There's white-labeled products. 781.96s - 783.48s | Insecure applications. 783.48s - 786.20s | Some of the primary vulnerabilities we've had 786.20s - 788.80s | in these devices have been app-based. 788.80s - 790.48s | It's the cool little app you use 790.48s - 792.56s | to log on to your solar power. 792.56s - 793.88s | Most of them are kind of junk. 793.88s - 795.96s | So here we are. 795.96s - 798.00s | There's not really any software bill of materials 798.00s - 799.08s | that comes with these things. 799.08s - 799.96s | And even if it did, 799.96s - 802.56s | we probably wouldn't trust it at this point. 802.56s - 805.80s | And again, there's really no vulnerability disclosure 805.80s - 807.52s | for the devices that are not made 807.52s - 808.84s | either in the US or Germany. 808.84s - 811.04s | We just don't have it because they're made in China. 811.04s - 815.80s | So what's more common in the Chinese-manufactured products? 815.80s - 819.88s | In particular, apart from the white-labeled products 819.88s - 821.52s | are not necessarily there. 821.52s - 823.72s | But these are things that are very, very common 823.72s - 826.56s | in the products that we're seeing being imported as well.


chunk_text:
vulnerabilities we've had 786.20s - 788.80s | in these devices have been app-based. 788.80s - 790.48s | It's the cool little app you use 790.48s - 792.56s | to log on to your solar power. 792.56s - 793.88s | Most of them are kind of junk. 793.88s - 795.96s | So here we are. 795.96s - 798.00s | There's not really any software bill of materials 798.00s - 799.08s | that comes with these things. 799.08s - 799.96s | And even if it did, 799.96s - 802.56s | we probably wouldn't trust it at this point. 802.56s - 805.80s | And again, there's really no vulnerability disclosure 805.80s - 807.52s | for the devices that are not made 807.52s - 808.84s | either in the US or Germany. 808.84s - 811.04s | We just don't have it because they're made in China. 811.04s - 815.80s | So what's more common in the Chinese-manufactured products? 815.80s - 819.88s | In particular, apart from the white-labeled products 819.88s - 821.52s | are not necessarily there. 821.52s - 823.72s | But these are things that are very, very common 823.72s - 826.56s | in the products that we're seeing being imported as well. 826.56s - 828.48s | And that's a challenge. 828.48s - 830.72s | These things are difficult to fix as well 830.72s - 832.32s | because a lot of those vulnerabilities, 833.08s - 835.16s | even if we find them, who's gonna fix them? 835.16s - 837.84s | So we're kind of in this interesting point. 837.84s - 840.68s | The hard-coded password thing has been coming up a lot 840.68s - 843.00s | and it's something we really need to fix. 843.00s - 845.56s | But the models are a big challenge. 846.52s - 849.20s | We like to say there have been no events 849.20s - 850.80s | on renewable energy infrastructure 850.80s - 852.76s | that have caused the power to go out. 852.76s - 854.40s | So there have been no cyber events 854.40s - 855.64s | that have caused a power outage. 855.64s - 856.48s | There have been events 856.48s - 858.20s | that have caused a communications outage 858.20s - 861.44s | or denial of service on communications 861.44s - 863.52s | or loss of visibility. 863.52s - 865.80s | There's also been a number of ransomware events 865.80s - 868.56s | starting all


chunk_text:
826.56s - 828.48s | And that's a challenge. 828.48s - 830.72s | These things are difficult to fix as well 830.72s - 832.32s | because a lot of those vulnerabilities, 833.08s - 835.16s | even if we find them, who's gonna fix them? 835.16s - 837.84s | So we're kind of in this interesting point. 837.84s - 840.68s | The hard-coded password thing has been coming up a lot 840.68s - 843.00s | and it's something we really need to fix. 843.00s - 845.56s | But the models are a big challenge. 846.52s - 849.20s | We like to say there have been no events 849.20s - 850.80s | on renewable energy infrastructure 850.80s - 852.76s | that have caused the power to go out. 852.76s - 854.40s | So there have been no cyber events 854.40s - 855.64s | that have caused a power outage. 855.64s - 856.48s | There have been events 856.48s - 858.20s | that have caused a communications outage 858.20s - 861.44s | or denial of service on communications 861.44s - 863.52s | or loss of visibility. 863.52s - 865.80s | There's also been a number of ransomware events 865.80s - 868.56s | starting all the way back in 2014. 868.56s - 871.60s | There's been a number of sort of ransomware, 871.60s - 874.40s | different wind turbine events also. 874.40s - 876.24s | Again, ransomware is bad, 876.24s - 878.60s | but it's not necessarily turned the power off. 878.60s - 881.08s | Nobody hit the big red button. 881.08s - 883.16s | Some of the more interesting ones in the US, 883.16s - 886.08s | there was a company that had a denial of service attack 886.08s - 888.60s | in 2019 there in Utah. 888.60s - 891.20s | Very, very large manufacturer 891.20s - 893.48s | or sort of very large owner of wind power. 893.48s - 895.12s | They had a denial of service attack 895.12s - 896.68s | that essentially took down comms. 896.68s - 898.44s | They couldn't see it. 898.44s - 899.96s | When you lose comms to renewable energy, 899.96s - 901.96s | most things will stay online. 901.96s - 903.64s | They won't necessarily fail, 903.64s - 905.12s | but over time they will start to fail. 905.12s - 907.20s | So you do still need to get that back. 907.20s - 908.66s | These


chunk_text:
the way back in 2014. 868.56s - 871.60s | There's been a number of sort of ransomware, 871.60s - 874.40s | different wind turbine events also. 874.40s - 876.24s | Again, ransomware is bad, 876.24s - 878.60s | but it's not necessarily turned the power off. 878.60s - 881.08s | Nobody hit the big red button. 881.08s - 883.16s | Some of the more interesting ones in the US, 883.16s - 886.08s | there was a company that had a denial of service attack 886.08s - 888.60s | in 2019 there in Utah. 888.60s - 891.20s | Very, very large manufacturer 891.20s - 893.48s | or sort of very large owner of wind power. 893.48s - 895.12s | They had a denial of service attack 895.12s - 896.68s | that essentially took down comms. 896.68s - 898.44s | They couldn't see it. 898.44s - 899.96s | When you lose comms to renewable energy, 899.96s - 901.96s | most things will stay online. 901.96s - 903.64s | They won't necessarily fail, 903.64s - 905.12s | but over time they will start to fail. 905.12s - 907.20s | So you do still need to get that back. 907.20s - 908.66s | These things are happening. 908.66s - 909.90s | We do have challenges. 911.36s - 914.08s | But some key points from a policy perspective, 914.08s - 915.68s | we've put a lot of responsibility 915.68s - 918.60s | for securing these devices on the average consumer. 918.60s - 920.24s | So if you have an inverter 920.24s - 922.00s | or solar installation in your house 922.00s - 923.70s | or battery installation, 923.70s - 924.88s | generally you're the consumer 924.88s - 927.68s | or you're the one responsible for updating the system. 927.68s - 929.80s | We're also putting them everywhere. 929.80s - 931.84s | The example here, we are putting them on 931.84s - 935.04s | lots of low income and disadvantaged community locations, 935.04s - 937.08s | multifamily housing. 937.08s - 937.92s | That's a challenge. 937.92s - 938.88s | Who's gonna update that? 938.88s - 940.04s | Who's in charge of it? 940.04s - 940.88s | Who's doing that? 940.88s - 943.16s | We're saying this is to help people with their energy bills, 943.16s - 944.94s | but we're also then giving them products 944.98s - 947.26s | that potentially have security vulnerabilities 947.26s -


chunk_text:
things are happening. 908.66s - 909.90s | We do have challenges. 911.36s - 914.08s | But some key points from a policy perspective, 914.08s - 915.68s | we've put a lot of responsibility 915.68s - 918.60s | for securing these devices on the average consumer. 918.60s - 920.24s | So if you have an inverter 920.24s - 922.00s | or solar installation in your house 922.00s - 923.70s | or battery installation, 923.70s - 924.88s | generally you're the consumer 924.88s - 927.68s | or you're the one responsible for updating the system. 927.68s - 929.80s | We're also putting them everywhere. 929.80s - 931.84s | The example here, we are putting them on 931.84s - 935.04s | lots of low income and disadvantaged community locations, 935.04s - 937.08s | multifamily housing. 937.08s - 937.92s | That's a challenge. 937.92s - 938.88s | Who's gonna update that? 938.88s - 940.04s | Who's in charge of it? 940.04s - 940.88s | Who's doing that? 940.88s - 943.16s | We're saying this is to help people with their energy bills, 943.16s - 944.94s | but we're also then giving them products 944.98s - 947.26s | that potentially have security vulnerabilities 947.26s - 949.10s | that cause them more challenges. 949.10s - 951.44s | But we put the responsibility on the consumer, 951.44s - 955.18s | whether these devices have one or 92 vulnerabilities in them, 955.18s - 956.46s | the person that's updating it 956.46s - 958.74s | is the person standing beside it that owns it, 958.74s - 961.66s | not necessarily the company connecting to it, 961.66s - 963.90s | but they're responsible for securing these devices 963.90s - 966.38s | because when there was an end phase 966.38s - 970.70s | had a pretty bad vulnerability that came out 970.70s - 973.10s | and in the instructions of how to fix it, 973.10s - 974.78s | it was essentially the consumers need to go 975.62s - 976.46s | and do something and reboot it. 976.46s - 978.10s | So it was a very long list of things 978.10s - 980.46s | that we were expecting, say, Joe down the road to do. 980.46s - 981.84s | No offense, Joe's in the room. 981.84s - 983.78s | There's like 14 of them. 983.78s - 985.30s | But we're expecting someone down the road 985.30s - 986.30s | to go and


chunk_text:
949.10s | that cause them more challenges. 949.10s - 951.44s | But we put the responsibility on the consumer, 951.44s - 955.18s | whether these devices have one or 92 vulnerabilities in them, 955.18s - 956.46s | the person that's updating it 956.46s - 958.74s | is the person standing beside it that owns it, 958.74s - 961.66s | not necessarily the company connecting to it, 961.66s - 963.90s | but they're responsible for securing these devices 963.90s - 966.38s | because when there was an end phase 966.38s - 970.70s | had a pretty bad vulnerability that came out 970.70s - 973.10s | and in the instructions of how to fix it, 973.10s - 974.78s | it was essentially the consumers need to go 975.62s - 976.46s | and do something and reboot it. 976.46s - 978.10s | So it was a very long list of things 978.10s - 980.46s | that we were expecting, say, Joe down the road to do. 980.46s - 981.84s | No offense, Joe's in the room. 981.84s - 983.78s | There's like 14 of them. 983.78s - 985.30s | But we're expecting someone down the road 985.30s - 986.30s | to go and fix all of that. 986.30s - 988.06s | And reading the analysis that came out, 988.06s - 990.98s | I was like, eh, I'm not sure anyone's gonna touch that. 990.98s - 992.26s | I think that's just gonna go 992.26s - 994.10s | and everyone's gonna leave it alone. 995.30s - 996.98s | But I talked a lot about 996.98s - 999.04s | sort of what the IT level attacks look like, 999.04s - 1001.24s | but the industrial control system landscape 1001.24s - 1002.94s | for a supply chain style attack 1002.98s - 1005.66s | looks a lot more like what happened with SolarWinds. 1005.66s - 1008.46s | I made these slides before CrowdStrike, so we're clear. 1008.46s - 1013.22s | But it looks a lot like what happened with SolarWinds. 1013.22s - 1016.50s | Like pushing out mass firmware to these devices 1016.50s - 1018.46s | is actually a pretty useful feature. 1018.46s - 1020.88s | I mentioned the customer's responsible. 1020.88s - 1022.46s | Pushing it out to millions of devices 1022.46s - 1023.94s | is actually pretty useful. 1023.94s - 1025.58s | It's had problems. 1025.58s - 1028.74s | There was a mass push out


chunk_text:
fix all of that. 986.30s - 988.06s | And reading the analysis that came out, 988.06s - 990.98s | I was like, eh, I'm not sure anyone's gonna touch that. 990.98s - 992.26s | I think that's just gonna go 992.26s - 994.10s | and everyone's gonna leave it alone. 995.30s - 996.98s | But I talked a lot about 996.98s - 999.04s | sort of what the IT level attacks look like, 999.04s - 1001.24s | but the industrial control system landscape 1001.24s - 1002.94s | for a supply chain style attack 1002.98s - 1005.66s | looks a lot more like what happened with SolarWinds. 1005.66s - 1008.46s | I made these slides before CrowdStrike, so we're clear. 1008.46s - 1013.22s | But it looks a lot like what happened with SolarWinds. 1013.22s - 1016.50s | Like pushing out mass firmware to these devices 1016.50s - 1018.46s | is actually a pretty useful feature. 1018.46s - 1020.88s | I mentioned the customer's responsible. 1020.88s - 1022.46s | Pushing it out to millions of devices 1022.46s - 1023.94s | is actually pretty useful. 1023.94s - 1025.58s | It's had problems. 1025.58s - 1028.74s | There was a mass push out to an island 1028.74s - 1030.70s | of a lot of inverters that came out. 1030.70s - 1032.02s | They pushed a firmware device out 1032.02s - 1034.02s | and everything stopped working for a minute. 1034.02s - 1035.10s | That wasn't great. 1036.06s - 1038.62s | But software events have happened. 1038.62s - 1041.70s | When there was an event in California recently, 1041.70s - 1043.54s | the batteries weren't working properly 1043.54s - 1045.86s | during a heat event, effectively. 1045.86s - 1047.26s | One of the things they did to fix it 1047.26s - 1048.50s | was a hot software fix, 1048.50s - 1050.10s | where they pushed out a new software 1050.10s - 1051.14s | to the device immediately 1051.14s - 1052.94s | and got it to start responding properly 1052.94s - 1054.46s | and start working again. 1054.46s - 1057.66s | Great, but that is also a potential vector 1057.66s - 1060.06s | for problems to come, as we all now know. 1061.06s - 1062.66s | But like I said, technically, 1062.66s - 1064.86s | what makes them a risk is a benefit. 1064.86s - 1068.42s | The other issue on our battery and inverter


chunk_text:
to an island 1028.74s - 1030.70s | of a lot of inverters that came out. 1030.70s - 1032.02s | They pushed a firmware device out 1032.02s - 1034.02s | and everything stopped working for a minute. 1034.02s - 1035.10s | That wasn't great. 1036.06s - 1038.62s | But software events have happened. 1038.62s - 1041.70s | When there was an event in California recently, 1041.70s - 1043.54s | the batteries weren't working properly 1043.54s - 1045.86s | during a heat event, effectively. 1045.86s - 1047.26s | One of the things they did to fix it 1047.26s - 1048.50s | was a hot software fix, 1048.50s - 1050.10s | where they pushed out a new software 1050.10s - 1051.14s | to the device immediately 1051.14s - 1052.94s | and got it to start responding properly 1052.94s - 1054.46s | and start working again. 1054.46s - 1057.66s | Great, but that is also a potential vector 1057.66s - 1060.06s | for problems to come, as we all now know. 1061.06s - 1062.66s | But like I said, technically, 1062.66s - 1064.86s | what makes them a risk is a benefit. 1064.86s - 1068.42s | The other issue on our battery and inverter supply chain 1068.42s - 1072.34s | is that the batteries are actually getting better over time. 1072.34s - 1075.74s | So we don't really make them here, like I've said, 1075.74s - 1077.14s | but they're getting better. 1077.14s - 1078.26s | That's the problem. 1078.26s - 1080.66s | If we look at quality of the batteries over time, 1080.66s - 1082.78s | if we look at the fires that have happened, 1082.78s - 1084.10s | they're pretty well-published. 1084.10s - 1086.06s | There's a number of fires, 1086.06s - 1087.70s | thermal runaways when batteries 1087.70s - 1089.54s | just go out of control and fire, 1089.94s - 1092.54s | potentially, is an electrochemical reaction. 1092.54s - 1094.50s | There's been less, if we look at all the data 1094.50s - 1095.94s | that's been collected. 1095.94s - 1097.62s | I think it's since 2018 1097.62s - 1099.86s | that's decreased drastically, actually. 1099.86s - 1101.34s | Partially, that's because, 1101.34s - 1104.38s | and there's been some news on this recently, 1104.38s - 1106.78s | Chinese companies have put lots and lots of money 1106.78s - 1109.06s | into improving the quality of their products. 1109.06s - 1111.56s


chunk_text:
supply chain 1068.42s - 1072.34s | is that the batteries are actually getting better over time. 1072.34s - 1075.74s | So we don't really make them here, like I've said, 1075.74s - 1077.14s | but they're getting better. 1077.14s - 1078.26s | That's the problem. 1078.26s - 1080.66s | If we look at quality of the batteries over time, 1080.66s - 1082.78s | if we look at the fires that have happened, 1082.78s - 1084.10s | they're pretty well-published. 1084.10s - 1086.06s | There's a number of fires, 1086.06s - 1087.70s | thermal runaways when batteries 1087.70s - 1089.54s | just go out of control and fire, 1089.94s - 1092.54s | potentially, is an electrochemical reaction. 1092.54s - 1094.50s | There's been less, if we look at all the data 1094.50s - 1095.94s | that's been collected. 1095.94s - 1097.62s | I think it's since 2018 1097.62s - 1099.86s | that's decreased drastically, actually. 1099.86s - 1101.34s | Partially, that's because, 1101.34s - 1104.38s | and there's been some news on this recently, 1104.38s - 1106.78s | Chinese companies have put lots and lots of money 1106.78s - 1109.06s | into improving the quality of their products. 1109.06s - 1111.56s | People are being educated in doing that. 1111.56s - 1113.64s | They also collect data from all of these. 1113.64s - 1116.10s | We like to say that you shouldn't be communicating 1116.10s - 1117.74s | with that, but they do collect data 1117.78s - 1119.70s | from all of the devices 1119.70s - 1121.38s | and look at the quality over time. 1121.38s - 1122.38s | What that actually means is 1122.38s - 1124.98s | there's a secondary communication device, for one, 1124.98s - 1127.90s | but also that they are improving their product. 1127.90s - 1129.50s | If we were actually looking to get rid 1129.50s - 1131.36s | of these batteries entirely, 1131.36s - 1132.94s | the only thing that's really gonna happen 1132.94s - 1135.50s | is these companies will go to a cheaper 1135.50s - 1138.22s | battery manufacturer or somebody else 1138.22s - 1139.28s | that's not on this list 1139.28s - 1141.38s | and end up potentially with more fires 1141.38s - 1144.34s | because they've not put as much money into it. 1144.34s - 1145.86s | We're kind of in this catch-22, 1145.86s - 1147.10s | just now of damned if


chunk_text:
| People are being educated in doing that. 1111.56s - 1113.64s | They also collect data from all of these. 1113.64s - 1116.10s | We like to say that you shouldn't be communicating 1116.10s - 1117.74s | with that, but they do collect data 1117.78s - 1119.70s | from all of the devices 1119.70s - 1121.38s | and look at the quality over time. 1121.38s - 1122.38s | What that actually means is 1122.38s - 1124.98s | there's a secondary communication device, for one, 1124.98s - 1127.90s | but also that they are improving their product. 1127.90s - 1129.50s | If we were actually looking to get rid 1129.50s - 1131.36s | of these batteries entirely, 1131.36s - 1132.94s | the only thing that's really gonna happen 1132.94s - 1135.50s | is these companies will go to a cheaper 1135.50s - 1138.22s | battery manufacturer or somebody else 1138.22s - 1139.28s | that's not on this list 1139.28s - 1141.38s | and end up potentially with more fires 1141.38s - 1144.34s | because they've not put as much money into it. 1144.34s - 1145.86s | We're kind of in this catch-22, 1145.86s - 1147.10s | just now of damned if we do, 1147.10s - 1148.70s | damned if we don't, one way or the other, 1148.70s - 1150.38s | and that's today. 1150.38s - 1152.34s | So what, who cares? 1152.34s - 1154.28s | We are dependent on these. 1154.28s - 1157.38s | There's some bad, bad, there's bad communications, 1157.38s - 1161.90s | hard-coded passwords, and questionable software 1161.90s - 1164.10s | used to update things. 1164.10s - 1166.66s | There's an over-concentration of Chinese products. 1166.66s - 1168.14s | It's a global issue. 1168.14s - 1170.26s | I don't necessarily think if anyone was targeting 1170.26s - 1171.54s | one particular supply chain, 1171.54s - 1173.34s | it would just affect this country. 1173.34s - 1174.60s | It would affect many countries 1174.64s - 1177.72s | because those products are global at this point. 1177.72s - 1181.00s | And we have fairly minimal control over the supply chain. 1181.00s - 1183.92s | Even if we do actually ban a number of products, 1183.92s - 1185.12s | it's not necessarily, 1185.12s - 1186.72s | you can't necessarily tell private industry, 1186.72s - 1187.76s | you will never buy this. 1187.76s - 1189.62s | That didn't work with some of


chunk_text:
we do, 1147.10s - 1148.70s | damned if we don't, one way or the other, 1148.70s - 1150.38s | and that's today. 1150.38s - 1152.34s | So what, who cares? 1152.34s - 1154.28s | We are dependent on these. 1154.28s - 1157.38s | There's some bad, bad, there's bad communications, 1157.38s - 1161.90s | hard-coded passwords, and questionable software 1161.90s - 1164.10s | used to update things. 1164.10s - 1166.66s | There's an over-concentration of Chinese products. 1166.66s - 1168.14s | It's a global issue. 1168.14s - 1170.26s | I don't necessarily think if anyone was targeting 1170.26s - 1171.54s | one particular supply chain, 1171.54s - 1173.34s | it would just affect this country. 1173.34s - 1174.60s | It would affect many countries 1174.64s - 1177.72s | because those products are global at this point. 1177.72s - 1181.00s | And we have fairly minimal control over the supply chain. 1181.00s - 1183.92s | Even if we do actually ban a number of products, 1183.92s - 1185.12s | it's not necessarily, 1185.12s - 1186.72s | you can't necessarily tell private industry, 1186.72s - 1187.76s | you will never buy this. 1187.76s - 1189.62s | That didn't work with some of the other products. 1189.62s - 1191.76s | So we don't necessarily have a ton of control 1191.76s - 1192.76s | over what people buy 1192.76s - 1194.88s | because we do have a free market here. 1194.88s - 1197.84s | So we've got minimal control over that, really. 1198.64s - 1200.76s | Again, I'm a failure mode person. 1200.76s - 1202.48s | I've done a lot of work on setting things on fire, 1202.48s - 1204.54s | so I have to talk about that for a minute. 1204.88s - 1206.10s | What would actually happen 1206.10s - 1208.98s | if there were a major attack on a battery? 1208.98s - 1211.30s | Well, I mentioned loss of visibility. 1211.30s - 1213.74s | This is the first thing that's probably the easiest 1213.74s - 1214.96s | and has happened. 1214.96s - 1217.30s | So essentially, if I was looking at difficulty, 1217.30s - 1219.22s | loss of communication and loss of visibility 1219.22s - 1222.68s | is kind of the biggest, easiest thing that can happen. 1222.68s - 1224.14s | Again, that's when they lose status 1224.14s - 1226.02s | monitoring our physical access alarms, 1226.02s - 1227.96s | which can


chunk_text:
the other products. 1189.62s - 1191.76s | So we don't necessarily have a ton of control 1191.76s - 1192.76s | over what people buy 1192.76s - 1194.88s | because we do have a free market here. 1194.88s - 1197.84s | So we've got minimal control over that, really. 1198.64s - 1200.76s | Again, I'm a failure mode person. 1200.76s - 1202.48s | I've done a lot of work on setting things on fire, 1202.48s - 1204.54s | so I have to talk about that for a minute. 1204.88s - 1206.10s | What would actually happen 1206.10s - 1208.98s | if there were a major attack on a battery? 1208.98s - 1211.30s | Well, I mentioned loss of visibility. 1211.30s - 1213.74s | This is the first thing that's probably the easiest 1213.74s - 1214.96s | and has happened. 1214.96s - 1217.30s | So essentially, if I was looking at difficulty, 1217.30s - 1219.22s | loss of communication and loss of visibility 1219.22s - 1222.68s | is kind of the biggest, easiest thing that can happen. 1222.68s - 1224.14s | Again, that's when they lose status 1224.14s - 1226.02s | monitoring our physical access alarms, 1226.02s - 1227.96s | which can be a bigger challenge. 1227.96s - 1230.04s | Like I said, there have been examples of this happening. 1230.04s - 1232.46s | The attack against the Viasat network 1232.46s - 1237.18s | during the Ukraine invasion in northern Europe 1237.18s - 1238.30s | actually resulted in the loss 1238.30s - 1241.26s | of about 6,000 wind turbines communication, 1241.26s - 1244.24s | which, you know, okay, they've lost comms, 1244.24s - 1245.38s | but they actually had to go out 1245.38s - 1247.66s | and fix each one of these individually in the end. 1247.66s - 1251.18s | So that was quite a big device problem, actually. 1252.38s - 1254.74s | Other types of potential attacks. 1254.74s - 1256.86s | If you look at a site, I mentioned fire. 1256.86s - 1258.36s | They are an electrochemical device. 1258.36s - 1259.30s | They do get hot. 1259.30s - 1261.74s | Please don't open them up to take a look at that. 1262.66s - 1265.30s | They might lose their preventive or proactive disconnect. 1265.30s - 1266.52s | Potentially, they stay online 1266.52s - 1268.50s | when you're trying to take them offline. 1268.50s - 1270.50s | There could be damage


chunk_text:
be a bigger challenge. 1227.96s - 1230.04s | Like I said, there have been examples of this happening. 1230.04s - 1232.46s | The attack against the Viasat network 1232.46s - 1237.18s | during the Ukraine invasion in northern Europe 1237.18s - 1238.30s | actually resulted in the loss 1238.30s - 1241.26s | of about 6,000 wind turbines communication, 1241.26s - 1244.24s | which, you know, okay, they've lost comms, 1244.24s - 1245.38s | but they actually had to go out 1245.38s - 1247.66s | and fix each one of these individually in the end. 1247.66s - 1251.18s | So that was quite a big device problem, actually. 1252.38s - 1254.74s | Other types of potential attacks. 1254.74s - 1256.86s | If you look at a site, I mentioned fire. 1256.86s - 1258.36s | They are an electrochemical device. 1258.36s - 1259.30s | They do get hot. 1259.30s - 1261.74s | Please don't open them up to take a look at that. 1262.66s - 1265.30s | They might lose their preventive or proactive disconnect. 1265.30s - 1266.52s | Potentially, they stay online 1266.52s - 1268.50s | when you're trying to take them offline. 1268.50s - 1270.50s | There could be damage to site capacity 1270.50s - 1271.66s | and environmental discharge. 1271.66s - 1272.50s | These things burn. 1272.50s - 1274.18s | If anyone was around here recently 1274.18s - 1277.38s | or tried to drive from California to LA, 1277.38s - 1279.68s | sorry, LA to Las Vegas, 1279.68s - 1281.74s | there was a huge lithium ion battery fire 1281.74s - 1283.90s | in the middle of this that shut down the road. 1283.90s - 1286.02s | And the plume from that was terrible. 1286.02s - 1287.94s | So chemicals are bad. 1287.94s - 1290.02s | Lastly, the most difficult style of these things 1290.02s - 1291.10s | is electric grid impact. 1291.10s - 1292.54s | It's the one we talk about most, 1292.54s - 1294.30s | but it's probably the most difficult. 1294.30s - 1296.58s | Again, to actually coordinate a massive event 1296.58s - 1298.50s | where the lights would go out everywhere, 1298.50s - 1299.90s | it would take a lot. 1299.90s - 1301.26s | It would take a lot of coordination 1301.26s - 1302.90s | with more expensive power electronics 1302.90s - 1305.50s | than we actually have, for one, which is good. 1305.50s -


chunk_text:
to site capacity 1270.50s - 1271.66s | and environmental discharge. 1271.66s - 1272.50s | These things burn. 1272.50s - 1274.18s | If anyone was around here recently 1274.18s - 1277.38s | or tried to drive from California to LA, 1277.38s - 1279.68s | sorry, LA to Las Vegas, 1279.68s - 1281.74s | there was a huge lithium ion battery fire 1281.74s - 1283.90s | in the middle of this that shut down the road. 1283.90s - 1286.02s | And the plume from that was terrible. 1286.02s - 1287.94s | So chemicals are bad. 1287.94s - 1290.02s | Lastly, the most difficult style of these things 1290.02s - 1291.10s | is electric grid impact. 1291.10s - 1292.54s | It's the one we talk about most, 1292.54s - 1294.30s | but it's probably the most difficult. 1294.30s - 1296.58s | Again, to actually coordinate a massive event 1296.58s - 1298.50s | where the lights would go out everywhere, 1298.50s - 1299.90s | it would take a lot. 1299.90s - 1301.26s | It would take a lot of coordination 1301.26s - 1302.90s | with more expensive power electronics 1302.90s - 1305.50s | than we actually have, for one, which is good. 1305.50s - 1307.22s | But it's still a challenge. 1307.22s - 1309.82s | We can't have these things misbehaving over the system. 1309.82s - 1311.14s | It's a bit of a problem. 1311.14s - 1313.18s | Okay, this is a policy talk, though. 1313.18s - 1315.30s | So how do we fix it? 1315.30s - 1316.56s | There are technical solutions. 1316.56s - 1318.30s | I'm not talking about those today. 1318.30s - 1320.46s | I'm mostly talking about, from a policy standpoint, 1320.82s - 1323.22s | what we really need to do to help fix these problems. 1323.22s - 1326.10s | I'd came up with four options of the direction we're going 1326.10s - 1328.34s | and the direction I think we should go. 1328.34s - 1331.02s | But so I was gonna talk about that a little bit. 1331.02s - 1332.46s | Option one and option two 1332.46s - 1334.86s | are really ban the companies, rip and replace. 1334.86s - 1336.24s | I mentioned before we've tried this. 1336.24s - 1338.06s | We are going down this path. 1338.06s - 1339.60s | We've already started the process. 1339.60s - 1341.86s | There's been 10 companies that have been banned


chunk_text:
1307.22s | But it's still a challenge. 1307.22s - 1309.82s | We can't have these things misbehaving over the system. 1309.82s - 1311.14s | It's a bit of a problem. 1311.14s - 1313.18s | Okay, this is a policy talk, though. 1313.18s - 1315.30s | So how do we fix it? 1315.30s - 1316.56s | There are technical solutions. 1316.56s - 1318.30s | I'm not talking about those today. 1318.30s - 1320.46s | I'm mostly talking about, from a policy standpoint, 1320.82s - 1323.22s | what we really need to do to help fix these problems. 1323.22s - 1326.10s | I'd came up with four options of the direction we're going 1326.10s - 1328.34s | and the direction I think we should go. 1328.34s - 1331.02s | But so I was gonna talk about that a little bit. 1331.02s - 1332.46s | Option one and option two 1332.46s - 1334.86s | are really ban the companies, rip and replace. 1334.86s - 1336.24s | I mentioned before we've tried this. 1336.24s - 1338.06s | We are going down this path. 1338.06s - 1339.60s | We've already started the process. 1339.60s - 1341.86s | There's been 10 companies that have been banned 1341.86s - 1344.66s | in the latest National Defense Authorization Act. 1344.66s - 1345.78s | That's gonna increase. 1345.78s - 1348.06s | We're looking at more and more. 1348.06s - 1349.82s | Fair enough, that's the direction we're going, 1350.18s - 1351.66s | but it doesn't mean we're gonna have a solution 1351.66s - 1353.82s | that works there because people might just go buy 1353.82s - 1356.06s | another cheaper one from the 70 list, 1356.06s - 1358.26s | so it might just get worse. 1358.26s - 1360.94s | But second option we're really looking at, 1360.94s - 1362.96s | rip and replace, that's came up. 1362.96s - 1364.02s | What does that mean? 1364.90s - 1367.50s | Around a couple, maybe three, four years ago 1367.50s - 1370.58s | when they banned Huawei from our networks, 1370.58s - 1374.58s | they also put funding in place for a bunch of small utilities 1374.58s - 1376.74s | to actually go replace all their Huawei equipment. 1376.74s - 1378.50s | It was an order that came through, 1378.50s - 1381.10s | and then they said, well, we're gonna put $4 billion 1381.10s - 1383.94s | towards small utilities taking out


chunk_text:
1341.86s - 1344.66s | in the latest National Defense Authorization Act. 1344.66s - 1345.78s | That's gonna increase. 1345.78s - 1348.06s | We're looking at more and more. 1348.06s - 1349.82s | Fair enough, that's the direction we're going, 1350.18s - 1351.66s | but it doesn't mean we're gonna have a solution 1351.66s - 1353.82s | that works there because people might just go buy 1353.82s - 1356.06s | another cheaper one from the 70 list, 1356.06s - 1358.26s | so it might just get worse. 1358.26s - 1360.94s | But second option we're really looking at, 1360.94s - 1362.96s | rip and replace, that's came up. 1362.96s - 1364.02s | What does that mean? 1364.90s - 1367.50s | Around a couple, maybe three, four years ago 1367.50s - 1370.58s | when they banned Huawei from our networks, 1370.58s - 1374.58s | they also put funding in place for a bunch of small utilities 1374.58s - 1376.74s | to actually go replace all their Huawei equipment. 1376.74s - 1378.50s | It was an order that came through, 1378.50s - 1381.10s | and then they said, well, we're gonna put $4 billion 1381.10s - 1383.94s | towards small utilities taking out this equipment, 1383.94s - 1386.46s | buying new equipment, not buying Huawei. 1386.46s - 1389.62s | That was great except we underfunded it by $3 billion. 1389.62s - 1392.74s | So we ended up with, I believe a couple months ago, 1392.74s - 1395.22s | $1 billion is spent out on doing this. 1395.22s - 1396.80s | Most of the utilities have not been able 1396.80s - 1399.26s | to actually replace the Huawei equipment 1399.26s - 1401.06s | because they can't buy it, 1401.06s - 1403.14s | and the funding isn't there for them to do it. 1403.14s - 1405.48s | So that didn't work. 1405.48s - 1407.82s | The other part is when we banned Huawei, 1407.82s - 1409.58s | they just went from being number two 1409.58s - 1410.92s | to number one in the world. 1410.92s - 1414.74s | That was the strangest part of this is when we banned them, 1414.74s - 1417.34s | and they actually did the market in 2019 in the US. 1417.34s - 1421.18s | In 2022, they were the top number one globally 1421.18s - 1422.78s | of inverter shipments in the world. 1422.78s - 1425.30s | So again, rip


chunk_text:
this equipment, 1383.94s - 1386.46s | buying new equipment, not buying Huawei. 1386.46s - 1389.62s | That was great except we underfunded it by $3 billion. 1389.62s - 1392.74s | So we ended up with, I believe a couple months ago, 1392.74s - 1395.22s | $1 billion is spent out on doing this. 1395.22s - 1396.80s | Most of the utilities have not been able 1396.80s - 1399.26s | to actually replace the Huawei equipment 1399.26s - 1401.06s | because they can't buy it, 1401.06s - 1403.14s | and the funding isn't there for them to do it. 1403.14s - 1405.48s | So that didn't work. 1405.48s - 1407.82s | The other part is when we banned Huawei, 1407.82s - 1409.58s | they just went from being number two 1409.58s - 1410.92s | to number one in the world. 1410.92s - 1414.74s | That was the strangest part of this is when we banned them, 1414.74s - 1417.34s | and they actually did the market in 2019 in the US. 1417.34s - 1421.18s | In 2022, they were the top number one globally 1421.18s - 1422.78s | of inverter shipments in the world. 1422.78s - 1425.30s | So again, rip and replace is not really, 1425.30s - 1428.54s | or even banning them is not necessarily the best beginning. 1429.94s - 1432.18s | Well, but now, again, if I only did the math 1432.18s - 1434.34s | on what we actually need to do 1434.34s - 1437.06s | to replace all the batteries in the system, 1437.06s - 1439.82s | at least in one state, it would be around $4 billion. 1439.82s - 1441.08s | So look at previous number 1441.08s - 1443.30s | for what we didn't fund the first time. 1443.30s - 1445.46s | I believe, I think in the US, I did the math, 1445.46s - 1448.26s | it was around $15 billion in the US 1448.26s - 1450.66s | to replace what we have currently installed or in line. 1450.66s - 1452.94s | So I don't think that's gonna happen. 1452.94s - 1454.46s | I don't know who's gonna push that bill, 1454.46s - 1456.18s | but it's not gonna. 1456.18s - 1458.82s | There's about 70 options, as it are 80 now. 1458.82s - 1461.14s | Again, they pop up from launched integration 1461.14s - 1462.18s | in about one year. 1462.18s - 1464.06s |


chunk_text:
and replace is not really, 1425.30s - 1428.54s | or even banning them is not necessarily the best beginning. 1429.94s - 1432.18s | Well, but now, again, if I only did the math 1432.18s - 1434.34s | on what we actually need to do 1434.34s - 1437.06s | to replace all the batteries in the system, 1437.06s - 1439.82s | at least in one state, it would be around $4 billion. 1439.82s - 1441.08s | So look at previous number 1441.08s - 1443.30s | for what we didn't fund the first time. 1443.30s - 1445.46s | I believe, I think in the US, I did the math, 1445.46s - 1448.26s | it was around $15 billion in the US 1448.26s - 1450.66s | to replace what we have currently installed or in line. 1450.66s - 1452.94s | So I don't think that's gonna happen. 1452.94s - 1454.46s | I don't know who's gonna push that bill, 1454.46s - 1456.18s | but it's not gonna. 1456.18s - 1458.82s | There's about 70 options, as it are 80 now. 1458.82s - 1461.14s | Again, they pop up from launched integration 1461.14s - 1462.18s | in about one year. 1462.18s - 1464.06s | That's just material. 1464.06s - 1465.86s | We'll end up playing whack-a-mole just forever. 1465.86s - 1467.86s | If you keep banning one, the next one will come up. 1467.86s - 1468.98s | We'll have to ban them. 1468.98s - 1469.86s | We'll just keep going 1469.86s - 1471.82s | until we have a really confused supply chain 1471.82s - 1473.58s | and no one knows who to buy it from. 1473.58s - 1475.94s | It'll also stop what we're doing currently. 1475.94s - 1477.98s | We will not be able to transition the grid 1477.98s - 1480.42s | if we keep going down this path. 1480.42s - 1482.14s | They do have a fire issue, like I said, 1482.14s - 1484.46s | but the ones that have caused more of the fires 1484.46s - 1486.70s | have not actually been on the ban list. 1486.70s - 1489.12s | So despite the fact they've had a natural physical impact 1489.12s - 1492.18s | of a piece of device or a fire, 1492.18s - 1494.58s | they're just not on a ban list yet for some reason. 1495.38s - 1497.78s | We're kind of stuck in these two worlds. 1497.78s -


chunk_text:
That's just material. 1464.06s - 1465.86s | We'll end up playing whack-a-mole just forever. 1465.86s - 1467.86s | If you keep banning one, the next one will come up. 1467.86s - 1468.98s | We'll have to ban them. 1468.98s - 1469.86s | We'll just keep going 1469.86s - 1471.82s | until we have a really confused supply chain 1471.82s - 1473.58s | and no one knows who to buy it from. 1473.58s - 1475.94s | It'll also stop what we're doing currently. 1475.94s - 1477.98s | We will not be able to transition the grid 1477.98s - 1480.42s | if we keep going down this path. 1480.42s - 1482.14s | They do have a fire issue, like I said, 1482.14s - 1484.46s | but the ones that have caused more of the fires 1484.46s - 1486.70s | have not actually been on the ban list. 1486.70s - 1489.12s | So despite the fact they've had a natural physical impact 1489.12s - 1492.18s | of a piece of device or a fire, 1492.18s - 1494.58s | they're just not on a ban list yet for some reason. 1495.38s - 1497.78s | We're kind of stuck in these two worlds. 1497.78s - 1500.54s | Again, I try not to be super negative, 1500.54s - 1503.14s | but there is progress, there is investment in the country. 1503.14s - 1504.26s | The Chips for America Act 1504.26s - 1506.14s | is bringing in a lot of the manufacture of chips, 1506.14s - 1509.58s | which will help solve some of our power electronics problems 1509.58s - 1512.78s | in a few years, probably not right now. 1512.78s - 1514.10s | Also, there is investment 1514.10s - 1516.50s | in battery cell material being built, 1516.50s - 1518.06s | which is all over the country. 1518.06s - 1520.34s | But again, we have five to 10 years 1520.34s - 1521.66s | before that's a reality. 1521.66s - 1523.54s | So we have to keep going, 1523.54s - 1524.78s | and that's not necessarily gonna work. 1524.78s - 1527.02s | We also, again, still need the demand signal. 1527.94s - 1529.38s | The other problem we have 1529.38s - 1532.10s | is actually inspecting the equipment. 1532.10s - 1534.14s | There's a lot of vulnerability analysis programs, 1534.14s - 1536.38s | that's great, but if I look at the list 1536.38s - 1538.66s | of


chunk_text:
1500.54s | Again, I try not to be super negative, 1500.54s - 1503.14s | but there is progress, there is investment in the country. 1503.14s - 1504.26s | The Chips for America Act 1504.26s - 1506.14s | is bringing in a lot of the manufacture of chips, 1506.14s - 1509.58s | which will help solve some of our power electronics problems 1509.58s - 1512.78s | in a few years, probably not right now. 1512.78s - 1514.10s | Also, there is investment 1514.10s - 1516.50s | in battery cell material being built, 1516.50s - 1518.06s | which is all over the country. 1518.06s - 1520.34s | But again, we have five to 10 years 1520.34s - 1521.66s | before that's a reality. 1521.66s - 1523.54s | So we have to keep going, 1523.54s - 1524.78s | and that's not necessarily gonna work. 1524.78s - 1527.02s | We also, again, still need the demand signal. 1527.94s - 1529.38s | The other problem we have 1529.38s - 1532.10s | is actually inspecting the equipment. 1532.10s - 1534.14s | There's a lot of vulnerability analysis programs, 1534.14s - 1536.38s | that's great, but if I look at the list 1536.38s - 1538.66s | of the inverters that have been analyzed, 1538.66s - 1543.66s | most of them aren't really reported through any US entity. 1544.50s - 1545.62s | We aren't necessarily saying 1545.62s - 1547.22s | that this is on the vulnerability list, 1547.22s - 1549.18s | here's what you need to fix. 1549.18s - 1551.22s | It's something from a policy perspective, 1551.22s - 1552.94s | I would love to see change. 1552.98s - 1555.14s | I believe that when we bring in products 1555.14s - 1557.74s | to this country to use on our critical infrastructure, 1557.74s - 1559.34s | we should be able to analyze it, 1559.34s - 1560.58s | we should be able to look inside it, 1560.58s - 1562.66s | and we should be able to push that out 1562.66s - 1564.18s | for other people to fix. 1564.18s - 1566.06s | It's a contract teeth issue 1566.06s - 1567.38s | that we kind of need at the moment. 1567.38s - 1569.26s | And I think I believe it needs to be pushed 1569.26s - 1570.22s | from the national level, 1570.22s - 1572.14s | so people can actually change their contracts, 1572.14s - 1574.50s | so we can


chunk_text:
the inverters that have been analyzed, 1538.66s - 1543.66s | most of them aren't really reported through any US entity. 1544.50s - 1545.62s | We aren't necessarily saying 1545.62s - 1547.22s | that this is on the vulnerability list, 1547.22s - 1549.18s | here's what you need to fix. 1549.18s - 1551.22s | It's something from a policy perspective, 1551.22s - 1552.94s | I would love to see change. 1552.98s - 1555.14s | I believe that when we bring in products 1555.14s - 1557.74s | to this country to use on our critical infrastructure, 1557.74s - 1559.34s | we should be able to analyze it, 1559.34s - 1560.58s | we should be able to look inside it, 1560.58s - 1562.66s | and we should be able to push that out 1562.66s - 1564.18s | for other people to fix. 1564.18s - 1566.06s | It's a contract teeth issue 1566.06s - 1567.38s | that we kind of need at the moment. 1567.38s - 1569.26s | And I think I believe it needs to be pushed 1569.26s - 1570.22s | from the national level, 1570.22s - 1572.14s | so people can actually change their contracts, 1572.14s - 1574.50s | so we can start doing this kind of work. 1574.50s - 1575.38s | But it's a huge challenge. 1575.38s - 1578.18s | We have the technology and we have the capability. 1578.18s - 1580.66s | We could keep these inverters and just analyze them, 1580.66s - 1582.18s | but we need to actually push back 1582.18s - 1583.62s | on the rules that we have in place 1583.62s - 1585.58s | for us not really looking at these devices 1585.58s - 1587.38s | or reporting it. 1587.38s - 1588.98s | Just, it'd be a deterrence, 1588.98s - 1589.94s | it'd be really good to actually say 1589.94s - 1591.14s | we're gonna analyze this equipment 1591.14s - 1593.10s | and we're gonna report vulnerabilities. 1593.10s - 1595.62s | But there's a trade-off there also. 1595.62s - 1596.46s | I'm speeding up, 1596.46s - 1598.34s | because I'm probably running out of time. 1598.34s - 1601.46s | But like I said, we have had analysis done 1601.46s - 1603.18s | on many of the common products, 1603.18s - 1604.42s | and that's happening. 1604.42s - 1606.18s | We need this for all of them, though. 1606.18s - 1608.54s | If we're gonna use all


chunk_text:
start doing this kind of work. 1574.50s - 1575.38s | But it's a huge challenge. 1575.38s - 1578.18s | We have the technology and we have the capability. 1578.18s - 1580.66s | We could keep these inverters and just analyze them, 1580.66s - 1582.18s | but we need to actually push back 1582.18s - 1583.62s | on the rules that we have in place 1583.62s - 1585.58s | for us not really looking at these devices 1585.58s - 1587.38s | or reporting it. 1587.38s - 1588.98s | Just, it'd be a deterrence, 1588.98s - 1589.94s | it'd be really good to actually say 1589.94s - 1591.14s | we're gonna analyze this equipment 1591.14s - 1593.10s | and we're gonna report vulnerabilities. 1593.10s - 1595.62s | But there's a trade-off there also. 1595.62s - 1596.46s | I'm speeding up, 1596.46s - 1598.34s | because I'm probably running out of time. 1598.34s - 1601.46s | But like I said, we have had analysis done 1601.46s - 1603.18s | on many of the common products, 1603.18s - 1604.42s | and that's happening. 1604.42s - 1606.18s | We need this for all of them, though. 1606.18s - 1608.54s | If we're gonna use all of these devices, 1608.54s - 1610.90s | we need the ability to analyze what's inside them 1610.90s - 1611.98s | and keep doing that. 1613.26s - 1614.90s | Just on the last part, 1614.90s - 1617.90s | we need to have better contract requirements for people. 1617.90s - 1619.30s | Again, this is policy talk, 1619.30s - 1622.50s | but contracts are one of our bigger problems. 1622.50s - 1623.98s | People write contracts 1623.98s - 1625.90s | that don't necessarily put into there, 1625.90s - 1626.94s | we're gonna look at your equipment, 1626.94s - 1631.42s | or we need to have particular TMZs for your firmware. 1631.42s - 1632.62s | We don't put in monitoring. 1632.62s - 1634.86s | We have to fix our contracting. 1634.86s - 1636.02s | But just lastly, 1636.02s - 1638.78s | the last one that keeps coming up is regulation. 1639.74s - 1640.58s | We're saying, 1640.58s - 1642.18s | oh, we're gonna regulate clean energy security. 1642.18s - 1644.42s | These companies will be under NRCSIP, 1644.42s - 1646.54s | which is the critical infrastructure protection rules 1646.54s - 1647.78s | in this country. 1647.78s - 1649.34s | I'm not sure who


chunk_text:
of these devices, 1608.54s - 1610.90s | we need the ability to analyze what's inside them 1610.90s - 1611.98s | and keep doing that. 1613.26s - 1614.90s | Just on the last part, 1614.90s - 1617.90s | we need to have better contract requirements for people. 1617.90s - 1619.30s | Again, this is policy talk, 1619.30s - 1622.50s | but contracts are one of our bigger problems. 1622.50s - 1623.98s | People write contracts 1623.98s - 1625.90s | that don't necessarily put into there, 1625.90s - 1626.94s | we're gonna look at your equipment, 1626.94s - 1631.42s | or we need to have particular TMZs for your firmware. 1631.42s - 1632.62s | We don't put in monitoring. 1632.62s - 1634.86s | We have to fix our contracting. 1634.86s - 1636.02s | But just lastly, 1636.02s - 1638.78s | the last one that keeps coming up is regulation. 1639.74s - 1640.58s | We're saying, 1640.58s - 1642.18s | oh, we're gonna regulate clean energy security. 1642.18s - 1644.42s | These companies will be under NRCSIP, 1644.42s - 1646.54s | which is the critical infrastructure protection rules 1646.54s - 1647.78s | in this country. 1647.78s - 1649.34s | I'm not sure who we're gonna regulate. 1649.34s - 1651.34s | So just in the last piece, 1651.34s - 1653.30s | or who is going to regulate them. 1654.94s - 1656.50s | There's one site I was looking at 1656.50s - 1659.06s | that had one developer and owner, 1659.06s - 1662.26s | they sold those site rights to somebody else 1662.26s - 1663.62s | who are finance bros. 1663.62s - 1664.82s | Sorry, that's my running joke, 1664.82s - 1666.10s | that they keep selling these sites 1666.10s - 1667.38s | to finance bro companies, 1667.38s - 1669.26s | which isn't necessarily the greatest idea 1669.26s - 1671.58s | when they own a gigawatt of generation. 1671.58s - 1672.42s | I don't know how many of them 1672.42s - 1674.14s | you want in charge of a gigawatt of generation, 1674.14s - 1675.66s | so we're clear. 1675.66s - 1676.98s | That site was developed. 1676.98s - 1680.02s | That owner then sold that again to another French company. 1680.02s - 1681.70s | So now we have a site 1681.70s - 1684.10s | that owns a gigawatt of generation in one state 1684.10s - 1687.38s | that is owned by a French company.


chunk_text:
we're gonna regulate. 1649.34s - 1651.34s | So just in the last piece, 1651.34s - 1653.30s | or who is going to regulate them. 1654.94s - 1656.50s | There's one site I was looking at 1656.50s - 1659.06s | that had one developer and owner, 1659.06s - 1662.26s | they sold those site rights to somebody else 1662.26s - 1663.62s | who are finance bros. 1663.62s - 1664.82s | Sorry, that's my running joke, 1664.82s - 1666.10s | that they keep selling these sites 1666.10s - 1667.38s | to finance bro companies, 1667.38s - 1669.26s | which isn't necessarily the greatest idea 1669.26s - 1671.58s | when they own a gigawatt of generation. 1671.58s - 1672.42s | I don't know how many of them 1672.42s - 1674.14s | you want in charge of a gigawatt of generation, 1674.14s - 1675.66s | so we're clear. 1675.66s - 1676.98s | That site was developed. 1676.98s - 1680.02s | That owner then sold that again to another French company. 1680.02s - 1681.70s | So now we have a site 1681.70s - 1684.10s | that owns a gigawatt of generation in one state 1684.10s - 1687.38s | that is owned by a French company. 1687.38s - 1688.50s | Okay, fair enough. 1688.50s - 1690.38s | It's physically connected to our grid though. 1690.38s - 1691.78s | So there's a utility over here 1691.78s - 1694.06s | that has the electrons flowing to it. 1694.06s - 1695.50s | And then there's the components 1695.50s - 1696.84s | that are manufactured in China. 1697.24s - 1698.96s | There's the third party operations and maintenance 1698.96s - 1700.04s | that's somewhere else. 1700.04s - 1701.80s | Then there's this fourth party operator 1701.80s - 1705.68s | somewhere over here doing something else, which is weird. 1705.68s - 1707.36s | And then they bid into the market. 1707.36s - 1708.64s | This one was in Texas, 1708.64s - 1710.56s | but they're still bidding into the market. 1710.56s - 1711.68s | So they have this connection 1711.68s - 1714.52s | to the operator for the state also. 1714.52s - 1716.12s | So I've been trying to work out for a while, 1716.12s - 1718.48s | which person is the regulated one? 1718.48s - 1720.60s | Who's registering as the generator here? 1720.60s - 1722.40s | Who's the one that follows the rules? 1722.40s - 1724.32s | I'm not


chunk_text:
1687.38s - 1688.50s | Okay, fair enough. 1688.50s - 1690.38s | It's physically connected to our grid though. 1690.38s - 1691.78s | So there's a utility over here 1691.78s - 1694.06s | that has the electrons flowing to it. 1694.06s - 1695.50s | And then there's the components 1695.50s - 1696.84s | that are manufactured in China. 1697.24s - 1698.96s | There's the third party operations and maintenance 1698.96s - 1700.04s | that's somewhere else. 1700.04s - 1701.80s | Then there's this fourth party operator 1701.80s - 1705.68s | somewhere over here doing something else, which is weird. 1705.68s - 1707.36s | And then they bid into the market. 1707.36s - 1708.64s | This one was in Texas, 1708.64s - 1710.56s | but they're still bidding into the market. 1710.56s - 1711.68s | So they have this connection 1711.68s - 1714.52s | to the operator for the state also. 1714.52s - 1716.12s | So I've been trying to work out for a while, 1716.12s - 1718.48s | which person is the regulated one? 1718.48s - 1720.60s | Who's registering as the generator here? 1720.60s - 1722.40s | Who's the one that follows the rules? 1722.40s - 1724.32s | I'm not sure which one it is at this point. 1724.32s - 1726.20s | So we need to work that out 1726.56s - 1729.16s | if we're gonna do anything from enforcing different things. 1729.16s - 1730.00s | I'll leave this up here. 1730.00s - 1730.84s | It was my last slide, 1730.84s - 1732.96s | but we will not have time to cover it. 1732.96s - 1734.92s | A while ago, we did run an exercise 1734.92s - 1737.24s | on the case I just said, 1737.24s - 1739.40s | but running an exercise of what if we essentially 1739.40s - 1741.36s | did an emergency mass push of firmware 1741.36s - 1744.36s | to millions of inverter devices at the same time 1744.36s - 1746.84s | and brick them all with a lot of people in the room 1746.84s - 1749.40s | trying to, not that that ever happens, 1749.40s - 1753.32s | but we ran this exercise trying to get some good answers 1753.32s - 1755.12s | from all of those people that have ownership 1755.12s - 1757.72s | to work out whose problem this really would be. 1757.72s - 1759.24s | And it was one of the more


chunk_text:
sure which one it is at this point. 1724.32s - 1726.20s | So we need to work that out 1726.56s - 1729.16s | if we're gonna do anything from enforcing different things. 1729.16s - 1730.00s | I'll leave this up here. 1730.00s - 1730.84s | It was my last slide, 1730.84s - 1732.96s | but we will not have time to cover it. 1732.96s - 1734.92s | A while ago, we did run an exercise 1734.92s - 1737.24s | on the case I just said, 1737.24s - 1739.40s | but running an exercise of what if we essentially 1739.40s - 1741.36s | did an emergency mass push of firmware 1741.36s - 1744.36s | to millions of inverter devices at the same time 1744.36s - 1746.84s | and brick them all with a lot of people in the room 1746.84s - 1749.40s | trying to, not that that ever happens, 1749.40s - 1753.32s | but we ran this exercise trying to get some good answers 1753.32s - 1755.12s | from all of those people that have ownership 1755.12s - 1757.72s | to work out whose problem this really would be. 1757.72s - 1759.24s | And it was one of the more confusing things 1759.24s - 1760.64s | we've ever had to do because everyone 1760.64s - 1762.84s | was just kind of staring at me with a, 1762.84s - 1764.68s | that doesn't seem good, that's not good. 1764.68s - 1766.00s | I was like, okay, great. 1766.00s - 1768.00s | So we won't have time to cover it, 1768.00s - 1770.00s | but these are the sort of questions we were asking 1770.00s - 1773.56s | and things we were thinking about here on what if we did, 1773.56s - 1775.16s | we called it the sun splat exercise, 1775.16s - 1776.76s | which I'm still pretty proud of, 1776.76s - 1779.64s | but it was essentially if what if we happened 1779.64s - 1782.28s | to mass push things and brick everything 1782.28s - 1783.56s | and who would fix it? 1783.56s - 1785.12s | With the idea that someone would actually have 1785.12s - 1788.26s | to manually go fix millions of devices at the same time. 1788.26s - 1789.56s | Again, that never happens. 1790.88s - 1794.00s | Well, that's my summary, but again, we need to do this. 1794.00s - 1795.28s | We're stuck between two


chunk_text:
confusing things 1759.24s - 1760.64s | we've ever had to do because everyone 1760.64s - 1762.84s | was just kind of staring at me with a, 1762.84s - 1764.68s | that doesn't seem good, that's not good. 1764.68s - 1766.00s | I was like, okay, great. 1766.00s - 1768.00s | So we won't have time to cover it, 1768.00s - 1770.00s | but these are the sort of questions we were asking 1770.00s - 1773.56s | and things we were thinking about here on what if we did, 1773.56s - 1775.16s | we called it the sun splat exercise, 1775.16s - 1776.76s | which I'm still pretty proud of, 1776.76s - 1779.64s | but it was essentially if what if we happened 1779.64s - 1782.28s | to mass push things and brick everything 1782.28s - 1783.56s | and who would fix it? 1783.56s - 1785.12s | With the idea that someone would actually have 1785.12s - 1788.26s | to manually go fix millions of devices at the same time. 1788.26s - 1789.56s | Again, that never happens. 1790.88s - 1794.00s | Well, that's my summary, but again, we need to do this. 1794.00s - 1795.28s | We're stuck between two worlds, 1795.28s - 1797.04s | but we have to get renewable energy 1797.04s - 1797.96s | and clean energy out there. 1797.96s - 1800.72s | It's just a fact, we do not want to keep going 1800.72s - 1802.44s | with climate change the way it is. 1802.44s - 1804.76s | We've got to fix this, it's a thing. 1804.76s - 1806.20s | Whether you believe in it or not, 1806.20s - 1808.24s | this is one of the solutions that should be happening 1808.24s - 1809.88s | and is going to happen. 1809.88s - 1813.24s | We do have adversarial entities in the supply chain. 1813.92s - 1814.96s | We need to somehow fix that also, 1814.96s - 1816.84s | but we have technology solutions that we can do. 1816.84s - 1818.88s | We just need a number of policy changes 1818.88s - 1822.48s | for us to be able to do the technology pieces that we have. 1822.48s - 1825.76s | We can fix it, but we've got to actually do the things 1825.76s - 1828.00s | and actually start doing the installations the right way 1828.00s - 1830.52s | and do the connections the right way for


chunk_text:
worlds, 1795.28s - 1797.04s | but we have to get renewable energy 1797.04s - 1797.96s | and clean energy out there. 1797.96s - 1800.72s | It's just a fact, we do not want to keep going 1800.72s - 1802.44s | with climate change the way it is. 1802.44s - 1804.76s | We've got to fix this, it's a thing. 1804.76s - 1806.20s | Whether you believe in it or not, 1806.20s - 1808.24s | this is one of the solutions that should be happening 1808.24s - 1809.88s | and is going to happen. 1809.88s - 1813.24s | We do have adversarial entities in the supply chain. 1813.92s - 1814.96s | We need to somehow fix that also, 1814.96s - 1816.84s | but we have technology solutions that we can do. 1816.84s - 1818.88s | We just need a number of policy changes 1818.88s - 1822.48s | for us to be able to do the technology pieces that we have. 1822.48s - 1825.76s | We can fix it, but we've got to actually do the things 1825.76s - 1828.00s | and actually start doing the installations the right way 1828.00s - 1830.52s | and do the connections the right way for future. 1830.52s - 1833.68s | But that's my summary for today. 1833.68s - 1836.40s | Thank you for listening and if there's any questions, 1836.40s - 1837.34s | I don't know if I have time, 1837.34s - 1839.40s | but I can answer at the end as well.


chunk_text:
future. 1830.52s - 1833.68s | But that's my summary for today. 1833.68s - 1836.40s | Thank you for listening and if there's any questions, 1836.40s - 1837.34s | I don't know if I have time, 1837.34s - 1839.40s | but I can answer at the end as well.


chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=6qTLxglm2fs", "title": "DEF CON 32 - Safecracking for Everyone - Jared Dygert", "description": "Learn the inner workings and vulnerabilities of mechanical combination safe locks! Safe manipulation is an underrated aspect of locksport and this talk will guide you through the ins and outs of how to do it.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1761, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.14s - 2.14s | This text was transcribed using whisper model: large-v2 All right, sorry for the delays 3.10s - 7.14s | So this is safe cracking for everyone. My name is Jared 13.63s - 15.95s | Okay, so a little bit about me. I 17.07s - 20.23s | Am a lock sport enthusiast. I have been picking locks 20.75s - 26.87s | For close to 20 years now and I got into safe manipulation shortly after 26.87s - 29.63s | I've been doing that for about 15 years. I 30.59s - 32.59s | game a bunch 32.59s - 37.99s | And I'm a professional rock climber. That's like my main source of income at the moment 38.39s - 43.63s | so this talk this talk will cover how 44.49s - 47.73s | specifically group two mechanical safe locks work 47.85s - 54.97s | so there's different classifications, which I will go over and group two is sort of just the standard sort of 55.37s - 57.45s | security classification for safe locks 58.81s - 62.57s | Whatever safe you buy essentially if it has a mechanical lock on it 63.09s - 66.69s | Vast majority of the time it will be a group two safe lock 66.69s - 72.45s | I will cover the flaws in this design and how we can exploit them as well as 72.73s - 74.93s | variations on the techniques that we use 76.69s - 80.41s | And then I will also cover differences between group two safe locks 80.85s - 86.49s | They all operate on the same principles, but there are some slight differences that set them apart from each other 87.09s - 94.90s | so the security classification starts with group two as the lowest security classification and 95.22s - 100.06s | There is absolutely nothing preventing anyone from figuring out the combination 100.90s - 108.66s | and then group 2m there are some small improvements to where they make it a lot harder to manipulate and 109.22s - 116.26s


chunk_text:
work 47.85s - 54.97s | so there's different classifications, which I will go over and group two is sort of just the standard sort of 55.37s - 57.45s | security classification for safe locks 58.81s - 62.57s | Whatever safe you buy essentially if it has a mechanical lock on it 63.09s - 66.69s | Vast majority of the time it will be a group two safe lock 66.69s - 72.45s | I will cover the flaws in this design and how we can exploit them as well as 72.73s - 74.93s | variations on the techniques that we use 76.69s - 80.41s | And then I will also cover differences between group two safe locks 80.85s - 86.49s | They all operate on the same principles, but there are some slight differences that set them apart from each other 87.09s - 94.90s | so the security classification starts with group two as the lowest security classification and 95.22s - 100.06s | There is absolutely nothing preventing anyone from figuring out the combination 100.90s - 108.66s | and then group 2m there are some small improvements to where they make it a lot harder to manipulate and 109.22s - 116.26s | then group one is the high security classification where they add in a lot of extra features and 117.70s - 122.14s | Supposedly they're supposed to withstand manipulation for about 20 hours minimum 123.02s - 125.02s | However, that is completely 125.74s - 127.74s | false and I 127.98s - 133.14s | Don't actually understand why they have different groups because they all kind of suck at the moment 133.38s - 138.62s | and then group 1r is the same as group 1, but the R stands for 140.18s - 144.58s | Radiologically resistant so a common attack could be 145.14s - 148.34s | using x-rays to view the inside of the lock and 148.54s - 155.82s | So they make the parts inside out of plastic usually Delrin a low-density plastic and it makes it really hard to x-ray it 156.26s - 160.40s | So I will cover the parts of a lock here 163.32s - 165.32s | normally, I would go up to the screen and 166.40s - 169.04s | Point, but however, I don't think that's possible 170.08s - 172.24s | So you should see a large 172.92s - 180.08s | Brass circle in the center


chunk_text:
| then group one is the high security classification where they add in a lot of extra features and 117.70s - 122.14s | Supposedly they're supposed to withstand manipulation for about 20 hours minimum 123.02s - 125.02s | However, that is completely 125.74s - 127.74s | false and I 127.98s - 133.14s | Don't actually understand why they have different groups because they all kind of suck at the moment 133.38s - 138.62s | and then group 1r is the same as group 1, but the R stands for 140.18s - 144.58s | Radiologically resistant so a common attack could be 145.14s - 148.34s | using x-rays to view the inside of the lock and 148.54s - 155.82s | So they make the parts inside out of plastic usually Delrin a low-density plastic and it makes it really hard to x-ray it 156.26s - 160.40s | So I will cover the parts of a lock here 163.32s - 165.32s | normally, I would go up to the screen and 166.40s - 169.04s | Point, but however, I don't think that's possible 170.08s - 172.24s | So you should see a large 172.92s - 180.08s | Brass circle in the center and it's got a bit of a cutout on it. That is called the drive cam 187.09s - 189.09s | All 207.64s - 210.52s | Right, cool. So here this is the drive cam 210.88s - 217.48s | This is connected directly with a dial. So as you turn the dial this drive cam will turn as well 218.56s - 222.76s | We have this arm. This arm is we just call it the the lever 223.64s - 230.08s | But this protrusion is called the nose and that nose as you can see rests on top of the drive cam 231.16s - 233.16s | Behind the drive cam we have wheels 235.08s - 249.45s | All right, so when you dial the combination on one of these locks what happens is these wheels 249.61s - 255.89s | They have a cutout called a gate in them. Okay, there's one gate per wheel and dialing the correct combination 256.77s - 259.21s | Lines up each of these gates with each other 259.81s - 264.85s | Now behind the lever you can see a bar. That bar is a fence. That's what we call it 264.85s - 270.45s | And when


chunk_text:
and it's got a bit of a cutout on it. That is called the drive cam 187.09s - 189.09s | All 207.64s - 210.52s | Right, cool. So here this is the drive cam 210.88s - 217.48s | This is connected directly with a dial. So as you turn the dial this drive cam will turn as well 218.56s - 222.76s | We have this arm. This arm is we just call it the the lever 223.64s - 230.08s | But this protrusion is called the nose and that nose as you can see rests on top of the drive cam 231.16s - 233.16s | Behind the drive cam we have wheels 235.08s - 249.45s | All right, so when you dial the combination on one of these locks what happens is these wheels 249.61s - 255.89s | They have a cutout called a gate in them. Okay, there's one gate per wheel and dialing the correct combination 256.77s - 259.21s | Lines up each of these gates with each other 259.81s - 264.85s | Now behind the lever you can see a bar. That bar is a fence. That's what we call it 264.85s - 270.45s | And when you dial the correct combination all of the gates will be lined up under the fence 270.45s - 273.93s | And that is so that the fence can fall into the gates 274.61s - 281.91s | so here we can see the correct combination has been dialed and 283.99s - 291.63s | When you turn the drive cam so that this cutout is underneath the nose that allows everything to fall down and what that does is 291.91s - 297.43s | Turning the dial further here pulls on the nose and allows the bolt to be retracted 297.83s - 310.22s | If I can get this camera to show at least somewhat I can show a little 310.62s - 312.02s | example here 312.02s - 319.35s | so we are looking at the top of the lock and you can see and 320.59s - 324.43s | You can see there is a bar and that is the fence 325.63s - 327.63s | Behind it underneath it are the wheels 328.75s - 330.75s | When you dial the combination 331.43s - 333.15s | Here we go 333.15s - 335.91s | Okay, it will line up each of the gates 336.99s -


chunk_text:
you dial the correct combination all of the gates will be lined up under the fence 270.45s - 273.93s | And that is so that the fence can fall into the gates 274.61s - 281.91s | so here we can see the correct combination has been dialed and 283.99s - 291.63s | When you turn the drive cam so that this cutout is underneath the nose that allows everything to fall down and what that does is 291.91s - 297.43s | Turning the dial further here pulls on the nose and allows the bolt to be retracted 297.83s - 310.22s | If I can get this camera to show at least somewhat I can show a little 310.62s - 312.02s | example here 312.02s - 319.35s | so we are looking at the top of the lock and you can see and 320.59s - 324.43s | You can see there is a bar and that is the fence 325.63s - 327.63s | Behind it underneath it are the wheels 328.75s - 330.75s | When you dial the combination 331.43s - 333.15s | Here we go 333.15s - 335.91s | Okay, it will line up each of the gates 336.99s - 338.11s | underneath 338.11s - 344.17s | that fence 344.17s - 346.69s | Sorry, it's a little hard to keep this in view 352.16s - 354.16s | Okay, so I 354.32s - 357.16s | Am lining up the gates underneath the fence 372.66s - 387.50s | Sorry for the technical difficulties here. Not sure how visible that is 388.50s - 393.36s | All right. Well, we'll just go back to the slides 394.20s - 407.18s | So essentially the way that it works is each of the wheels 407.86s - 409.50s | to in turn 409.50s - 415.06s | Grab the wheels behind it. So that drive cam which is directly connected to the dial 416.58s - 420.18s | Has a protrusion much like this wheel does here and 420.86s - 426.50s | That protrusion will ride in a groove on top of the wheel behind it 427.26s - 434.94s | until it hits the the protrusion on that wheel and so essentially it can freely ride around and 435.42s - 440.14s | Once it makes contact it will grab that wheel and it will turn with the dial 440.90s - 444.58s | That wheel on the bottom also has


chunk_text:
338.11s | underneath 338.11s - 344.17s | that fence 344.17s - 346.69s | Sorry, it's a little hard to keep this in view 352.16s - 354.16s | Okay, so I 354.32s - 357.16s | Am lining up the gates underneath the fence 372.66s - 387.50s | Sorry for the technical difficulties here. Not sure how visible that is 388.50s - 393.36s | All right. Well, we'll just go back to the slides 394.20s - 407.18s | So essentially the way that it works is each of the wheels 407.86s - 409.50s | to in turn 409.50s - 415.06s | Grab the wheels behind it. So that drive cam which is directly connected to the dial 416.58s - 420.18s | Has a protrusion much like this wheel does here and 420.86s - 426.50s | That protrusion will ride in a groove on top of the wheel behind it 427.26s - 434.94s | until it hits the the protrusion on that wheel and so essentially it can freely ride around and 435.42s - 440.14s | Once it makes contact it will grab that wheel and it will turn with the dial 440.90s - 444.58s | That wheel on the bottom also has a protrusion 444.58s - 452.18s | This is called a drive pin that drive pin will ride in a groove on the wheel behind it until it hits 452.94s - 457.82s | That wheel and picks it up. So it takes one rotation 458.62s - 461.82s | To ensure that you have picked up the next wheel 461.98s - 468.34s | That's because if you are moving the wheel in one direction and you reverse rotation 468.34s - 472.70s | You have to go all the way around until they can pick it up again 475.43s - 477.43s | Here's an example 477.43s - 478.47s | I 478.47s - 480.47s | Put this here in case the camera didn't work 480.79s - 485.87s | I would like to show this more visually, but this is the best I have at the moment 485.99s - 492.39s | So we can imagine this black piece at the back being the drive cam and each of these wheels being one of the wheels in 492.39s - 500.83s | The lock so what happens is this this drive cam will spin around and here it's making contact with the drive pin of this 500.91s


chunk_text:
a protrusion 444.58s - 452.18s | This is called a drive pin that drive pin will ride in a groove on the wheel behind it until it hits 452.94s - 457.82s | That wheel and picks it up. So it takes one rotation 458.62s - 461.82s | To ensure that you have picked up the next wheel 461.98s - 468.34s | That's because if you are moving the wheel in one direction and you reverse rotation 468.34s - 472.70s | You have to go all the way around until they can pick it up again 475.43s - 477.43s | Here's an example 477.43s - 478.47s | I 478.47s - 480.47s | Put this here in case the camera didn't work 480.79s - 485.87s | I would like to show this more visually, but this is the best I have at the moment 485.99s - 492.39s | So we can imagine this black piece at the back being the drive cam and each of these wheels being one of the wheels in 492.39s - 500.83s | The lock so what happens is this this drive cam will spin around and here it's making contact with the drive pin of this 500.91s - 507.35s | Third wheel the wheel closest to the drive cam is called the third wheel then the second and then the first and 507.67s - 511.67s | So if we wanted to spin this wheel and the other direction 512.23s - 518.55s | this drive cam would turn all the way around one full rotation until it can hit this drive pin and 519.43s - 524.39s | Same thing for this third wheel if we want the third wheel to pick up the second wheel 524.39s - 529.47s | It has to go all the way around until it hits the drive pin and so on and so forth 529.47s - 536.19s | And so this is how all the wheels are able to move together, but also be able to be set separately 536.83s - 541.78s | so vulnerabilities 542.38s - 545.22s | So here we have a close-up of the drive cam 545.78s - 550.14s | Now the nose normally rests on top of the drive drive cam 550.14s - 555.66s | It will rest and ride along the whole outer edge of the drive cam as the drive cam spins 556.66s - 563.54s | But


chunk_text:
- 507.35s | Third wheel the wheel closest to the drive cam is called the third wheel then the second and then the first and 507.67s - 511.67s | So if we wanted to spin this wheel and the other direction 512.23s - 518.55s | this drive cam would turn all the way around one full rotation until it can hit this drive pin and 519.43s - 524.39s | Same thing for this third wheel if we want the third wheel to pick up the second wheel 524.39s - 529.47s | It has to go all the way around until it hits the drive pin and so on and so forth 529.47s - 536.19s | And so this is how all the wheels are able to move together, but also be able to be set separately 536.83s - 541.78s | so vulnerabilities 542.38s - 545.22s | So here we have a close-up of the drive cam 545.78s - 550.14s | Now the nose normally rests on top of the drive drive cam 550.14s - 555.66s | It will rest and ride along the whole outer edge of the drive cam as the drive cam spins 556.66s - 563.54s | But when we get to this drop-in area the nose will drop in ever so slightly and 564.94s - 570.38s | The vulnerability here is this is not symmetrical this drop-in point is 571.38s - 575.46s | Asymmetrical and on one side we have a great curve to it 575.46s - 580.98s | And so you can see the further down we go in this drop-in point 581.22s - 586.94s | There is less wiggle room from one side to the other is significantly less than up high 586.94s - 589.58s | There's much more wiggle room up there 589.58s - 596.06s | And the thing about this is because that nose will go a little bit into the drop-in point 596.62s - 603.12s | When we turn the drive cam we can feel it hit the wall of that drop-in area 604.30s - 607.98s | We can actually feel that on the dial as a bit of resistance 608.26s - 615.30s | So we spin the dial so that this drop-in point is centered below the nose that nose is not touching the drive cam at all 615.30s - 619.78s | and if we turn a little further than the


chunk_text:
when we get to this drop-in area the nose will drop in ever so slightly and 564.94s - 570.38s | The vulnerability here is this is not symmetrical this drop-in point is 571.38s - 575.46s | Asymmetrical and on one side we have a great curve to it 575.46s - 580.98s | And so you can see the further down we go in this drop-in point 581.22s - 586.94s | There is less wiggle room from one side to the other is significantly less than up high 586.94s - 589.58s | There's much more wiggle room up there 589.58s - 596.06s | And the thing about this is because that nose will go a little bit into the drop-in point 596.62s - 603.12s | When we turn the drive cam we can feel it hit the wall of that drop-in area 604.30s - 607.98s | We can actually feel that on the dial as a bit of resistance 608.26s - 615.30s | So we spin the dial so that this drop-in point is centered below the nose that nose is not touching the drive cam at all 615.30s - 619.78s | and if we turn a little further than the nose will hit the edge as 620.22s - 627.06s | In this picture right here, and we can feel when that happens, so if that nose is up high in 627.62s - 629.62s | this contact or this 629.66s - 630.78s | drop-in area 630.78s - 637.94s | Then there will be a significant distance between the two points of contact on the walls of that drop-in area 637.94s - 639.94s | And if the nose is lower 640.62s - 642.46s | There will be less 642.46s - 644.46s | distance between feeling those points of contact 646.91s - 653.55s | Now the thing that determines how high or low that nose is in that area is the fence 653.77s - 657.61s | The fence here you can see is resting above the wheels 657.73s - 660.93s | So we have the nose that is on the drive cam right now 660.93s - 667.89s | And that is what's keeping that fence up, but when we turn the dial so the nose is over that drop-in area 668.85s - 672.09s | Then the fence will fall down and rest on the wheel pack 672.29s - 678.53s | That is essentially


chunk_text:
nose will hit the edge as 620.22s - 627.06s | In this picture right here, and we can feel when that happens, so if that nose is up high in 627.62s - 629.62s | this contact or this 629.66s - 630.78s | drop-in area 630.78s - 637.94s | Then there will be a significant distance between the two points of contact on the walls of that drop-in area 637.94s - 639.94s | And if the nose is lower 640.62s - 642.46s | There will be less 642.46s - 644.46s | distance between feeling those points of contact 646.91s - 653.55s | Now the thing that determines how high or low that nose is in that area is the fence 653.77s - 657.61s | The fence here you can see is resting above the wheels 657.73s - 660.93s | So we have the nose that is on the drive cam right now 660.93s - 667.89s | And that is what's keeping that fence up, but when we turn the dial so the nose is over that drop-in area 668.85s - 672.09s | Then the fence will fall down and rest on the wheel pack 672.29s - 678.53s | That is essentially testing for if the combination is correct or not if the gates were all aligned 678.61s - 684.87s | Then the fence would not rest on top of the wheel pack it would simply fall in and that would allow the lock to open 685.71s - 692.83s | So here you can see the wheels are all different sizes, that's because we are not able to manufacture 693.63s - 698.21s | Multiple wheels to be exactly the same size or to be perfectly circular 698.71s - 704.19s | So here we will always have one wheel that's larger than the others as you can see here 704.19s - 711.67s | It is the first wheel all the way to the left furthest from the drive cam. There's space here on the second and third wheel 712.31s - 718.23s | So if we were to turn the dial so that the nose is over that drop-in point this fence 718.67s - 725.39s | Will hit and rest only on this first wheel because it sticks up above the rest 726.39s - 731.75s | Now this is where the previous slide comes in handy with the narrower 732.15s - 736.67s | Space


chunk_text:
testing for if the combination is correct or not if the gates were all aligned 678.61s - 684.87s | Then the fence would not rest on top of the wheel pack it would simply fall in and that would allow the lock to open 685.71s - 692.83s | So here you can see the wheels are all different sizes, that's because we are not able to manufacture 693.63s - 698.21s | Multiple wheels to be exactly the same size or to be perfectly circular 698.71s - 704.19s | So here we will always have one wheel that's larger than the others as you can see here 704.19s - 711.67s | It is the first wheel all the way to the left furthest from the drive cam. There's space here on the second and third wheel 712.31s - 718.23s | So if we were to turn the dial so that the nose is over that drop-in point this fence 718.67s - 725.39s | Will hit and rest only on this first wheel because it sticks up above the rest 726.39s - 731.75s | Now this is where the previous slide comes in handy with the narrower 732.15s - 736.67s | Space the further down you go because if we were to feel 737.23s - 742.39s | Those contact points that point at which the nose hits the wall of the drop-in area 742.79s - 747.75s | Those are hard contact points and we measure how far apart those are 748.59s - 753.63s | We would get a certain number a certain reading. Let's say on the dial that is six increments 754.47s - 758.91s | But if we were to turn this first wheel and only this first wheel 759.19s - 765.11s | So that the gate is underneath the fence this fence will not be resting up as high anymore 765.11s - 767.91s | It will drop on to the next largest wheel 768.55s - 772.11s | Causing the nose to be further down in that contact area 773.03s - 780.23s | Which means that our points of contact are now going to be closer together instead of having a range of six 780.59s - 784.79s | Sorry six numbers on the dial apart. Maybe we can feel those points of resistance 785.75s - 792.91s | Five and a half numbers apart and that tells us that we have the


chunk_text:
the further down you go because if we were to feel 737.23s - 742.39s | Those contact points that point at which the nose hits the wall of the drop-in area 742.79s - 747.75s | Those are hard contact points and we measure how far apart those are 748.59s - 753.63s | We would get a certain number a certain reading. Let's say on the dial that is six increments 754.47s - 758.91s | But if we were to turn this first wheel and only this first wheel 759.19s - 765.11s | So that the gate is underneath the fence this fence will not be resting up as high anymore 765.11s - 767.91s | It will drop on to the next largest wheel 768.55s - 772.11s | Causing the nose to be further down in that contact area 773.03s - 780.23s | Which means that our points of contact are now going to be closer together instead of having a range of six 780.59s - 784.79s | Sorry six numbers on the dial apart. Maybe we can feel those points of resistance 785.75s - 792.91s | Five and a half numbers apart and that tells us that we have the largest wheel in the correct position 793.35s - 796.83s | So if we had let's say dialed a test combination and 797.27s - 802.11s | This is what we have and we measure the contact points to be six numbers apart 802.11s - 805.99s | And then we dial the same exact test combination again 806.43s - 812.39s | but with a different number for the first number in the combination which corresponds to the first wheel and that 812.71s - 817.71s | Places the gate underneath the fence and we see that decrease in the contact area 818.23s - 823.39s | Then we know whatever we just changed for that first number is the correct number 824.75s - 830.59s | So you can see here it is resting on top of the first wheel and first wheel only this fence is not 830.75s - 832.63s | Touching the second or third wheel 832.63s - 836.91s | So if that first wheel does rotate so that the gate is underneath it 836.91s - 841.93s | the fence will drop lower and we can feel that difference in the dial with how the 842.39s - 845.63s | Nose touches the sides


chunk_text:
largest wheel in the correct position 793.35s - 796.83s | So if we had let's say dialed a test combination and 797.27s - 802.11s | This is what we have and we measure the contact points to be six numbers apart 802.11s - 805.99s | And then we dial the same exact test combination again 806.43s - 812.39s | but with a different number for the first number in the combination which corresponds to the first wheel and that 812.71s - 817.71s | Places the gate underneath the fence and we see that decrease in the contact area 818.23s - 823.39s | Then we know whatever we just changed for that first number is the correct number 824.75s - 830.59s | So you can see here it is resting on top of the first wheel and first wheel only this fence is not 830.75s - 832.63s | Touching the second or third wheel 832.63s - 836.91s | So if that first wheel does rotate so that the gate is underneath it 836.91s - 841.93s | the fence will drop lower and we can feel that difference in the dial with how the 842.39s - 845.63s | Nose touches the sides of that contact area 846.15s - 854.00s | So essentially we have our plan of attack what we can do is we can find what's called a common low point and 856.20s - 862.48s | That's because the wheels are not necessarily larger or smaller than the other wheels they can be 863.28s - 868.52s | Bumpy or irregularly shaped. So if we have the wheels in the correct position 869.20s - 871.80s | Maybe there is a bump on another wheel 872.36s - 875.18s | Preventing that fence from falling down completely 875.36s - 882.20s | So we want to find a point where across every wheel is a common low point where there are no extra 882.48s - 889.68s | bumps or ridges on any of the wheels and we have that nose as far down in that contact area as we can and 890.76s - 895.12s | We do something called wheel isolation. We will isolate the third wheel 895.36s - 901.14s | So essentially if we are only moving one wheel at a time and we get a reading 901.38s - 906.90s | That the nose is lower in that contact area so we can feel


chunk_text:
of that contact area 846.15s - 854.00s | So essentially we have our plan of attack what we can do is we can find what's called a common low point and 856.20s - 862.48s | That's because the wheels are not necessarily larger or smaller than the other wheels they can be 863.28s - 868.52s | Bumpy or irregularly shaped. So if we have the wheels in the correct position 869.20s - 871.80s | Maybe there is a bump on another wheel 872.36s - 875.18s | Preventing that fence from falling down completely 875.36s - 882.20s | So we want to find a point where across every wheel is a common low point where there are no extra 882.48s - 889.68s | bumps or ridges on any of the wheels and we have that nose as far down in that contact area as we can and 890.76s - 895.12s | We do something called wheel isolation. We will isolate the third wheel 895.36s - 901.14s | So essentially if we are only moving one wheel at a time and we get a reading 901.38s - 906.90s | That the nose is lower in that contact area so we can feel that point of resistance 907.70s - 912.26s | where we feel the nose hitting the sides of that of that drive cam and 913.46s - 917.58s | We feel that decrease and only one wheel is moving 917.78s - 922.52s | We know exactly what number and what wheel is in the combination 922.52s - 927.42s | So if we're only moving the third wheel and we move it from 10 to 12 927.42s - 934.06s | And we have a drop in that in that reading that means 12 is a third number in the combination and makes things 934.62s - 936.62s | really simple and 936.90s - 941.34s | Then we have to look for gate signatures, which is that drop that I have been talking about 941.86s - 946.70s | Feeling the drop in the range of the of the two contact points 947.74s - 950.90s | Once we find that we can find the center of the gate 950.90s - 957.18s | We want to be precise because there there is a range at which the lock will open 957.42s - 962.42s | That is not the correct combination. If you set your combination to be


chunk_text:
that point of resistance 907.70s - 912.26s | where we feel the nose hitting the sides of that of that drive cam and 913.46s - 917.58s | We feel that decrease and only one wheel is moving 917.78s - 922.52s | We know exactly what number and what wheel is in the combination 922.52s - 927.42s | So if we're only moving the third wheel and we move it from 10 to 12 927.42s - 934.06s | And we have a drop in that in that reading that means 12 is a third number in the combination and makes things 934.62s - 936.62s | really simple and 936.90s - 941.34s | Then we have to look for gate signatures, which is that drop that I have been talking about 941.86s - 946.70s | Feeling the drop in the range of the of the two contact points 947.74s - 950.90s | Once we find that we can find the center of the gate 950.90s - 957.18s | We want to be precise because there there is a range at which the lock will open 957.42s - 962.42s | That is not the correct combination. If you set your combination to be 10 20 and 30 962.94s - 965.46s | Then you can open it with 9 966.38s - 967.38s | 21 967.38s - 974.46s | 29 that is a valid combination. Most locks will open plus or minus one and a quarter numbers on either side 975.42s - 979.90s | So we want to find the center as close to the true combination as we can 980.14s - 984.90s | In case we made a mistake along the way and we can repeat these steps for the second wheel 985.10s - 989.26s | So essentially we can move only the second wheel in the lock by itself 989.26s - 994.54s | And so if we find a gate signature if we find a decrease in that contact area 994.98s - 1001.18s | Then we know what our second number in the combination is and there is no need to do this for the first wheel 1001.18s - 1006.26s | And that's because if you have two numbers in a combination that has three numbers 1006.90s - 1011.22s | you just brute force the last one you just dial every two increments and 1011.62s - 1016.22s | Then you don't have to


chunk_text:
10 20 and 30 962.94s - 965.46s | Then you can open it with 9 966.38s - 967.38s | 21 967.38s - 974.46s | 29 that is a valid combination. Most locks will open plus or minus one and a quarter numbers on either side 975.42s - 979.90s | So we want to find the center as close to the true combination as we can 980.14s - 984.90s | In case we made a mistake along the way and we can repeat these steps for the second wheel 985.10s - 989.26s | So essentially we can move only the second wheel in the lock by itself 989.26s - 994.54s | And so if we find a gate signature if we find a decrease in that contact area 994.98s - 1001.18s | Then we know what our second number in the combination is and there is no need to do this for the first wheel 1001.18s - 1006.26s | And that's because if you have two numbers in a combination that has three numbers 1006.90s - 1011.22s | you just brute force the last one you just dial every two increments and 1011.62s - 1016.22s | Then you don't have to worry about trying to manipulate the lock and get a reading 1017.38s - 1024.98s | Now there's a reason we go from three to one and that's because the wheel that is closest to the drive cam is 1024.98s - 1026.66s | wheel three and 1026.66s - 1031.10s | That lever is spring-loaded. It has a spring pressing down on it 1031.66s - 1037.94s | That spring is only on one half. So it causes it to tilt. So when you are 1038.86s - 1045.06s | So when you are turning the dial so the nose is over that that drop-in point 1045.90s - 1053.10s | The fence will hit the wheels and it will actually tilt and cause it to only make contact with wheel three 1053.10s - 1060.42s | I want to say 90% of the time and I say 90% because it's not always the same due to tolerances 1060.62s - 1067.06s | But that is also why we find the common low point. That is why we find an area across all the wheels 1067.94s - 1071.62s | That don't interfere with each other that there are no bumps is that it will help us 1071.86s


chunk_text:
worry about trying to manipulate the lock and get a reading 1017.38s - 1024.98s | Now there's a reason we go from three to one and that's because the wheel that is closest to the drive cam is 1024.98s - 1026.66s | wheel three and 1026.66s - 1031.10s | That lever is spring-loaded. It has a spring pressing down on it 1031.66s - 1037.94s | That spring is only on one half. So it causes it to tilt. So when you are 1038.86s - 1045.06s | So when you are turning the dial so the nose is over that that drop-in point 1045.90s - 1053.10s | The fence will hit the wheels and it will actually tilt and cause it to only make contact with wheel three 1053.10s - 1060.42s | I want to say 90% of the time and I say 90% because it's not always the same due to tolerances 1060.62s - 1067.06s | But that is also why we find the common low point. That is why we find an area across all the wheels 1067.94s - 1071.62s | That don't interfere with each other that there are no bumps is that it will help us 1071.86s - 1076.18s | Ensure that wheel three will be the first wheel that the fence touches 1076.70s - 1082.64s | so the way we find this common low point is we can spin every wheel together and 1083.60s - 1085.20s | We can take 1085.20s - 1092.38s | Contact point readings, which means you go to the place at which you can feel that resistance which you can feel that nose 1092.84s - 1096.40s | hitting the side of that drive drive cam in that in that 1097.36s - 1105.16s | Contact area when you feel that resistance whatever number that is on the dial that is your contact point reading 1107.36s - 1108.64s | after that 1108.64s - 1114.04s | So so we pick up all the wheels and we take contact point readings every ten increments 1114.04s - 1118.96s | What that means is we will move every wheel together and on the dial 1118.96s - 1125.08s | We will stop at zero then we will turn to that contact area and get our contact point 1125.12s - 1131.92s | We will then turn all the wheels to ten and then twenty and then thirty so every ten


chunk_text:
- 1076.18s | Ensure that wheel three will be the first wheel that the fence touches 1076.70s - 1082.64s | so the way we find this common low point is we can spin every wheel together and 1083.60s - 1085.20s | We can take 1085.20s - 1092.38s | Contact point readings, which means you go to the place at which you can feel that resistance which you can feel that nose 1092.84s - 1096.40s | hitting the side of that drive drive cam in that in that 1097.36s - 1105.16s | Contact area when you feel that resistance whatever number that is on the dial that is your contact point reading 1107.36s - 1108.64s | after that 1108.64s - 1114.04s | So so we pick up all the wheels and we take contact point readings every ten increments 1114.04s - 1118.96s | What that means is we will move every wheel together and on the dial 1118.96s - 1125.08s | We will stop at zero then we will turn to that contact area and get our contact point 1125.12s - 1131.92s | We will then turn all the wheels to ten and then twenty and then thirty so every ten increments 1132.04s - 1139.68s | We take a contact point reading and we find the lowest reading we find the reading at which the wheels 1140.88s - 1145.68s | Read the lowest so that is when the nose will be furthest down in that drive cam 1146.80s - 1154.02s | We can leave the first and second wheels there so we can turn the dial until only the first two wheels are moving 1155.40s - 1158.64s | and leave them there actually the third one moves too, but 1159.40s - 1166.44s | we can it this gets the first and second wheels out of the way and we can then proceed to move the third wheel as 1166.56s - 1171.84s | We see fit that third wheel is closest to the drive cam. So it is the first to move 1171.96s - 1174.68s | So if we have our first and second wheels 1175.44s - 1179.52s | Somewhere we can move the third wheel without interrupting them 1179.92s - 1183.40s | So with the third wheel 1183.40s - 1189.84s | we can then spin and pick up the third wheel after we drop the first two off where they need


chunk_text:
increments 1132.04s - 1139.68s | We take a contact point reading and we find the lowest reading we find the reading at which the wheels 1140.88s - 1145.68s | Read the lowest so that is when the nose will be furthest down in that drive cam 1146.80s - 1154.02s | We can leave the first and second wheels there so we can turn the dial until only the first two wheels are moving 1155.40s - 1158.64s | and leave them there actually the third one moves too, but 1159.40s - 1166.44s | we can it this gets the first and second wheels out of the way and we can then proceed to move the third wheel as 1166.56s - 1171.84s | We see fit that third wheel is closest to the drive cam. So it is the first to move 1171.96s - 1174.68s | So if we have our first and second wheels 1175.44s - 1179.52s | Somewhere we can move the third wheel without interrupting them 1179.92s - 1183.40s | So with the third wheel 1183.40s - 1189.84s | we can then spin and pick up the third wheel after we drop the first two off where they need to be and 1190.36s - 1196.16s | We can move it every two increments so we can take that third wheel and let's say we can start on zero 1196.44s - 1203.32s | okay, and then we take a contact point reading so we just turn to that drop-in point and feel for the bit of resistance and 1203.32s - 1208.12s | See what numbers on the dial and then we can go back to where we left the third wheel from zero 1208.12s - 1210.12s | We can then turn to two 1210.20s - 1215.52s | Go take a contact point reading and then go turn our wheel to four take a contact point reading 1215.52s - 1218.28s | so we do this 50 times around the dial and 1219.04s - 1224.70s | Because we're only moving one wheel whenever we see a drop in our contact point reading 1225.44s - 1232.34s | We know that number we just turned the third wheel to is our third number in the combination. There is no ambiguity about it 1233.34s - 1239.98s | From there we can actually go back and just move the wheel through that area again 1239.98s -


chunk_text:
to be and 1190.36s - 1196.16s | We can move it every two increments so we can take that third wheel and let's say we can start on zero 1196.44s - 1203.32s | okay, and then we take a contact point reading so we just turn to that drop-in point and feel for the bit of resistance and 1203.32s - 1208.12s | See what numbers on the dial and then we can go back to where we left the third wheel from zero 1208.12s - 1210.12s | We can then turn to two 1210.20s - 1215.52s | Go take a contact point reading and then go turn our wheel to four take a contact point reading 1215.52s - 1218.28s | so we do this 50 times around the dial and 1219.04s - 1224.70s | Because we're only moving one wheel whenever we see a drop in our contact point reading 1225.44s - 1232.34s | We know that number we just turned the third wheel to is our third number in the combination. There is no ambiguity about it 1233.34s - 1239.98s | From there we can actually go back and just move the wheel through that area again 1239.98s - 1246.20s | So if we find a drop in that contact point when the third wheel is at 30 or let's say 32 1246.58s - 1249.50s | We can go back and because we went every two increments 1249.50s - 1255.58s | we can hit all the odd numbers and we can find the exact center of where it needs to be if 1256.66s - 1258.62s | We do not have a gate indication 1258.62s - 1265.30s | that's actually fine with this method because we can use the lowest point and pretend that is our actual number and 1265.46s - 1272.20s | Simply repeating these steps with the lowest points will actually give us the real the real number in the combinations 1272.76s - 1276.43s | The second wheel is a little more 1276.89s - 1283.13s | Difficult. Um, I am kind of skimming through this because my camera is not working. Unfortunately. I can't show this visually 1283.97s - 1290.77s | But the second wheel is just you you turn different number of times, but we're doing the same process 1291.13s - 1296.69s | We are finding a way to move only the second wheel by itself inside


chunk_text:
1246.20s | So if we find a drop in that contact point when the third wheel is at 30 or let's say 32 1246.58s - 1249.50s | We can go back and because we went every two increments 1249.50s - 1255.58s | we can hit all the odd numbers and we can find the exact center of where it needs to be if 1256.66s - 1258.62s | We do not have a gate indication 1258.62s - 1265.30s | that's actually fine with this method because we can use the lowest point and pretend that is our actual number and 1265.46s - 1272.20s | Simply repeating these steps with the lowest points will actually give us the real the real number in the combinations 1272.76s - 1276.43s | The second wheel is a little more 1276.89s - 1283.13s | Difficult. Um, I am kind of skimming through this because my camera is not working. Unfortunately. I can't show this visually 1283.97s - 1290.77s | But the second wheel is just you you turn different number of times, but we're doing the same process 1291.13s - 1296.69s | We are finding a way to move only the second wheel by itself inside the lock 1296.85s - 1299.97s | So when we get an indication from the dial 1300.85s - 1310.46s | We know it is the second number in the combination and for the first wheel if we had two good 1311.30s - 1312.26s | Indications 1312.26s - 1314.50s | Then we can brute force the first wheel 1315.10s - 1319.86s | Dialing zero then the second number and third number and then to second number third number 1319.86s - 1321.98s | We're just dialing full combinations 1321.98s - 1327.86s | And what that does is allow us to cut down on the amount of time we need we spend dialing 1327.94s - 1331.42s | Since going by every two numbers will open the lock 1333.06s - 1340.22s | We can also keep track of the lowest point still just in case we were wrong earlier if we just keep 1340.46s - 1346.50s | Intending that the lowest contact point readings we get is the real number in the combination 1346.50s - 1349.22s | we can keep on repeating this process and 1350.18s - 1360.22s | Eventually those low points will become the real numbers. So there are different types of group


chunk_text:
the lock 1296.85s - 1299.97s | So when we get an indication from the dial 1300.85s - 1310.46s | We know it is the second number in the combination and for the first wheel if we had two good 1311.30s - 1312.26s | Indications 1312.26s - 1314.50s | Then we can brute force the first wheel 1315.10s - 1319.86s | Dialing zero then the second number and third number and then to second number third number 1319.86s - 1321.98s | We're just dialing full combinations 1321.98s - 1327.86s | And what that does is allow us to cut down on the amount of time we need we spend dialing 1327.94s - 1331.42s | Since going by every two numbers will open the lock 1333.06s - 1340.22s | We can also keep track of the lowest point still just in case we were wrong earlier if we just keep 1340.46s - 1346.50s | Intending that the lowest contact point readings we get is the real number in the combination 1346.50s - 1349.22s | we can keep on repeating this process and 1350.18s - 1360.22s | Eventually those low points will become the real numbers. So there are different types of group two locks 1361.68s - 1363.68s | One second. Oh 1369.65s - 1371.85s | Sorry, just trying to make sure I don't go over time here 1373.33s - 1375.41s | So there are different types of group two locks 1375.77s - 1378.49s | Legar 3330 is a very common one 1378.49s - 1384.73s | It is not one I suggest starting out with the wheels are more oval. And what that means is 1385.45s - 1391.57s | We want our gates to be of each wheel under the fence and the lock. That is what tells us 1391.57s - 1395.01s | What is the right number or not with over wheels? 1395.53s - 1397.53s | One wheel may shadow 1398.01s - 1402.57s | Another wheel meaning when we have a gate underneath the fence 1402.57s - 1408.01s | Another wheel could have a large bump on it because it's oval and might be 1408.57s - 1411.05s | Keeping the fence from falling into that gate 1412.81s - 1415.21s | We also have die bolt it's a very common 1416.33s - 1424.73s | Lock for ATMs banks their drive cam is u-shaped. It is more symmetrical than what you have seen and


chunk_text:
two locks 1361.68s - 1363.68s | One second. Oh 1369.65s - 1371.85s | Sorry, just trying to make sure I don't go over time here 1373.33s - 1375.41s | So there are different types of group two locks 1375.77s - 1378.49s | Legar 3330 is a very common one 1378.49s - 1384.73s | It is not one I suggest starting out with the wheels are more oval. And what that means is 1385.45s - 1391.57s | We want our gates to be of each wheel under the fence and the lock. That is what tells us 1391.57s - 1395.01s | What is the right number or not with over wheels? 1395.53s - 1397.53s | One wheel may shadow 1398.01s - 1402.57s | Another wheel meaning when we have a gate underneath the fence 1402.57s - 1408.01s | Another wheel could have a large bump on it because it's oval and might be 1408.57s - 1411.05s | Keeping the fence from falling into that gate 1412.81s - 1415.21s | We also have die bolt it's a very common 1416.33s - 1424.73s | Lock for ATMs banks their drive cam is u-shaped. It is more symmetrical than what you have seen and 1425.29s - 1428.01s | So in that case we can actually take 1428.05s - 1434.29s | And so in that case we can actually take both contact points very carefully and 1434.89s - 1436.89s | That will give us 1436.93s - 1438.93s | the same results 1439.37s - 1444.01s | Here it's a little off-screen, but this is the most common model that you will see 1444.85s - 1446.69s | sergeant in Greenleaf 1446.69s - 1449.69s | 6730 or 6741 they are virtually identical 1450.45s - 1456.61s | One can just be opened with one and a half numbers on either side. So a little easier to open 1457.21s - 1461.43s | But then essentially they're the same and this is what I suggest everyone to start with 1462.61s - 1467.97s | After this talk, I will have a table with a bunch of these set up in the lock picking village 1468.81s - 1470.81s | And you can try hands-on 1471.53s - 1475.72s | so 1475.72s - 1482.88s | These are really important tips. Okay. These are not just tips if you don't succeed. These are things you have to go in 1483.56s - 1489.36s | Knowing and the


chunk_text:
1425.29s - 1428.01s | So in that case we can actually take 1428.05s - 1434.29s | And so in that case we can actually take both contact points very carefully and 1434.89s - 1436.89s | That will give us 1436.93s - 1438.93s | the same results 1439.37s - 1444.01s | Here it's a little off-screen, but this is the most common model that you will see 1444.85s - 1446.69s | sergeant in Greenleaf 1446.69s - 1449.69s | 6730 or 6741 they are virtually identical 1450.45s - 1456.61s | One can just be opened with one and a half numbers on either side. So a little easier to open 1457.21s - 1461.43s | But then essentially they're the same and this is what I suggest everyone to start with 1462.61s - 1467.97s | After this talk, I will have a table with a bunch of these set up in the lock picking village 1468.81s - 1470.81s | And you can try hands-on 1471.53s - 1475.72s | so 1475.72s - 1482.88s | These are really important tips. Okay. These are not just tips if you don't succeed. These are things you have to go in 1483.56s - 1489.36s | Knowing and the most important is being precise in everything not just dialing 1489.68s - 1496.44s | But reading the dial from the same angle every time if you look at it from the side that will change what you see on 1496.44s - 1498.44s | the dial 1498.56s - 1502.76s | You want to read the contact point with the same amount of force so we can feel 1503.24s - 1506.32s | When that nose hits the drive cam and that's our contact point 1506.80s - 1514.04s | However, we have to feel that with the same amount of force every time or we will get a different number on the dial 1515.16s - 1517.96s | we can also try with a known combination and 1518.80s - 1520.80s | What that means is 1521.12s - 1524.48s | Knowing the combination. So if I know the combination is 10 20 30 1524.48s - 1533.16s | I can only test from 0 to 40 for instance, and I know I should have an indication in that area 1533.16s - 1535.16s | So I do not have to waste my time 1535.36s - 1539.48s | With numbers that I know are not in the combination.


chunk_text:
most important is being precise in everything not just dialing 1489.68s - 1496.44s | But reading the dial from the same angle every time if you look at it from the side that will change what you see on 1496.44s - 1498.44s | the dial 1498.56s - 1502.76s | You want to read the contact point with the same amount of force so we can feel 1503.24s - 1506.32s | When that nose hits the drive cam and that's our contact point 1506.80s - 1514.04s | However, we have to feel that with the same amount of force every time or we will get a different number on the dial 1515.16s - 1517.96s | we can also try with a known combination and 1518.80s - 1520.80s | What that means is 1521.12s - 1524.48s | Knowing the combination. So if I know the combination is 10 20 30 1524.48s - 1533.16s | I can only test from 0 to 40 for instance, and I know I should have an indication in that area 1533.16s - 1535.16s | So I do not have to waste my time 1535.36s - 1539.48s | With numbers that I know are not in the combination. And so that helps 1540.20s - 1544.24s | Helps with getting a feel for what you are looking for 1545.24s - 1550.56s | You can also remove the back cover and look at inside the lock while you do it 1550.56s - 1553.52s | And I have removed all the back covers on the locks 1553.64s - 1558.96s | That will be for demonstration in the lock picking village so you can look inside and see what's happening. I 1559.80s - 1562.32s | Apologize. I had some technical difficulties 1562.52s - 1567.32s | So I understand that things are a little little complex with how I'm describing it 1567.32s - 1571.40s | But we do have in-person demonstrations at the lock picking village 1572.36s - 1574.56s | And if all else fails I do have a book 1575.96s - 1580.80s | So the book is safe cracking for everyone on Amazon, but you don't need to buy it 1580.80s - 1584.12s | That is only if you want the hardcopy version 1584.40s - 1591.00s | There is a free PDF if you search the title of the book on YouTube safe cracking for everyone 1591.48s - 1593.96s | There is


chunk_text:
And so that helps 1540.20s - 1544.24s | Helps with getting a feel for what you are looking for 1545.24s - 1550.56s | You can also remove the back cover and look at inside the lock while you do it 1550.56s - 1553.52s | And I have removed all the back covers on the locks 1553.64s - 1558.96s | That will be for demonstration in the lock picking village so you can look inside and see what's happening. I 1559.80s - 1562.32s | Apologize. I had some technical difficulties 1562.52s - 1567.32s | So I understand that things are a little little complex with how I'm describing it 1567.32s - 1571.40s | But we do have in-person demonstrations at the lock picking village 1572.36s - 1574.56s | And if all else fails I do have a book 1575.96s - 1580.80s | So the book is safe cracking for everyone on Amazon, but you don't need to buy it 1580.80s - 1584.12s | That is only if you want the hardcopy version 1584.40s - 1591.00s | There is a free PDF if you search the title of the book on YouTube safe cracking for everyone 1591.48s - 1593.96s | There is it's like nine parts 1593.96s - 1600.72s | But the first video in that series part one will have a link to the PDF of the book for free 1601.68s - 1605.68s | other great resources are also the lock picking subreddit and 1606.92s - 1612.16s | LPU lock pickers United discord. There should be a link to that discord in the subreddit 1612.48s - 1616.44s | There's also a game called Sophie safe cracking simulator 1616.44s - 1618.32s | I I don't know who Sophie is 1618.32s - 1623.48s | But this is the best safe cracking game as that's all it is 1623.48s - 1627.56s | it is just a lock and you can turn the dial and 1628.08s - 1635.08s | There's different tools in this game where you can have x-ray vision, you know, you can practice with all these different 1636.08s - 1639.40s | Variables that you can set up that you can't do in real life 1640.08s - 1643.76s | and of course the best resource is practice, um, I 1644.80s - 1646.80s | teach at a locksmithing 1647.48s - 1653.92s | University and I have my safe manipulation course is about 18 hours


chunk_text:
it's like nine parts 1593.96s - 1600.72s | But the first video in that series part one will have a link to the PDF of the book for free 1601.68s - 1605.68s | other great resources are also the lock picking subreddit and 1606.92s - 1612.16s | LPU lock pickers United discord. There should be a link to that discord in the subreddit 1612.48s - 1616.44s | There's also a game called Sophie safe cracking simulator 1616.44s - 1618.32s | I I don't know who Sophie is 1618.32s - 1623.48s | But this is the best safe cracking game as that's all it is 1623.48s - 1627.56s | it is just a lock and you can turn the dial and 1628.08s - 1635.08s | There's different tools in this game where you can have x-ray vision, you know, you can practice with all these different 1636.08s - 1639.40s | Variables that you can set up that you can't do in real life 1640.08s - 1643.76s | and of course the best resource is practice, um, I 1644.80s - 1646.80s | teach at a locksmithing 1647.48s - 1653.92s | University and I have my safe manipulation course is about 18 hours long 1654.92s - 1659.24s | That is not enough time for someone to be proficient with 1659.68s - 1663.72s | guidance someone can open their first lock in that time, but I 1664.12s - 1668.60s | Mean, I recommend at least 50 hours before you are even 1669.36s - 1675.80s | Proficient in in this field. The hardest part is learning once you know and understand how a lock works 1676.48s - 1682.00s | Then it's easy. It's just understanding that and getting the information that takes time 1684.32s - 1686.80s | Here is the direct link to the book 1687.36s - 1689.36s | I just hosted on Google Drive 1689.84s - 1693.76s | Hopefully this QR code shows up well enough, but this should take you directly 1694.48s - 1697.60s | To the PDF and you can download it and read it. I 1698.48s - 1703.48s | Do include information on high security safe manipulation 1703.68s - 1709.00s | This book as far as I know is the only place of that information 1709.00s - 1714.12s | You cannot find that anywhere else and by high security safe manipulation 1714.12s - 1717.48s | I mean there are locks in


chunk_text:
long 1654.92s - 1659.24s | That is not enough time for someone to be proficient with 1659.68s - 1663.72s | guidance someone can open their first lock in that time, but I 1664.12s - 1668.60s | Mean, I recommend at least 50 hours before you are even 1669.36s - 1675.80s | Proficient in in this field. The hardest part is learning once you know and understand how a lock works 1676.48s - 1682.00s | Then it's easy. It's just understanding that and getting the information that takes time 1684.32s - 1686.80s | Here is the direct link to the book 1687.36s - 1689.36s | I just hosted on Google Drive 1689.84s - 1693.76s | Hopefully this QR code shows up well enough, but this should take you directly 1694.48s - 1697.60s | To the PDF and you can download it and read it. I 1698.48s - 1703.48s | Do include information on high security safe manipulation 1703.68s - 1709.00s | This book as far as I know is the only place of that information 1709.00s - 1714.12s | You cannot find that anywhere else and by high security safe manipulation 1714.12s - 1717.48s | I mean there are locks in the group one category 1718.04s - 1722.44s | that have been around this these designs have been around for a while and 1723.08s - 1726.24s | Their methods of opening them are not new 1727.60s - 1731.32s | And they are very insecure and they are not publicly available 1731.80s - 1738.00s | so I decided to make that publicly available and this is the only place that I know of at which you can get that information and 1742.22s - 1746.34s | That is that is it. Thank you for coming. Um, I 1747.02s - 1751.66s | Really apologize for the technical difficulties and the convoluted explanations 1751.98s - 1759.54s | But we will have in-person demonstrations. You can try yourself at the lock-picking village immediately after this. Thank you


chunk_text:
the group one category 1718.04s - 1722.44s | that have been around this these designs have been around for a while and 1723.08s - 1726.24s | Their methods of opening them are not new 1727.60s - 1731.32s | And they are very insecure and they are not publicly available 1731.80s - 1738.00s | so I decided to make that publicly available and this is the only place that I know of at which you can get that information and 1742.22s - 1746.34s | That is that is it. Thank you for coming. Um, I 1747.02s - 1751.66s | Really apologize for the technical difficulties and the convoluted explanations 1751.98s - 1759.54s | But we will have in-person demonstrations. You can try yourself at the lock-picking village immediately after this. Thank you


chunk_text:
{ "webpage_url": "https://www.youtube.com/watch?v=42OjQtrJQ5s", "title": "DEF CON 32 - Small Satellite Modeling and Defender Software - Kyle Murbach", "description": "The proliferation of ride-share rocket launches and decrease in the overall cost of sending payloads to space due to recent successes in the private space industry has made small satellite systems a cost effective and time-efficient method to put research vehicles in space.\n\nThe University of Alabama in Huntsville\u2019s Center for Cybersecurity Research and Education (CCRE) has been funded by the U.S. Army Space and Missile Defense Command (SMDC) over the last several years to investigate the overall cybersecurity posture of small satellite systems. Numerous iterations of student teams led by CCRE and SMDC staff members have managed to accomplish notable research milestones.\n\nThis talk is meant to inform the next generation in aerospace cybersecurity by discussing our major research milestones, relevant findings, lessons learned, and areas of concern relating to the overall cybersecurity posture of small satellite systems.\n\nRelevant items to be covered in this talk include what it took to build a working small satellite system model as close to real-world as possible (Raspberry Pis vs PyCubed boards vs other contenders), implementation of small satellite functions (payload camera, radio communications, positioning/sensor array, orbital simulation, battery/solar charging, etc.), performing vulnerability analysis against the implemented model, creating different attack scenarios (MitM, DoS, spoofing, hardware attacks), implementing defensive mitigations (hardening scripts, command validation, health checks), and the development of a lightweight software solution named \u201cSmall Satellite Defender\u201d (SSD) designed to protect satellites from potential threat vectors.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1480, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 2.88s | This text was transcribed using whisper model: large-v2 All right, good afternoon, everyone. I'm going to go ahead and get started. 3.84s - 8.96s | So I'm here today to talk to you about small satellite modeling and defender software. 8.96s - 14.80s | My name is Dr. Kyle Merbach, and I'm from the University of Alabama in Huntsville Center for 14.80s - 20.08s | Cybersecurity Research and Education. And this project has been in partnership with the U.S. 20.08s - 24.16s | Army Space and Missile Defense Command, and we're excited to share with you what we've been doing 24.16s - 32.40s | the last few years. So a little bit about myself. I'm a principal research engineer at UAH CCRE. 32.40s - 38.88s | I've been there since 2020. I've been a PI


chunk_text:
simulation, battery/solar charging, etc.), performing vulnerability analysis against the implemented model, creating different attack scenarios (MitM, DoS, spoofing, hardware attacks), implementing defensive mitigations (hardening scripts, command validation, health checks), and the development of a lightweight software solution named \u201cSmall Satellite Defender\u201d (SSD) designed to protect satellites from potential threat vectors.", "channel_url": "https://www.youtube.com/channel/UC6Om9kAkl32dWlDSNlDS9Iw", "duration": 1480, "channel": "DEFCONConference", "uploader": "DEFCONConference", "upload_date": "20241016" } 0.00s - 2.88s | This text was transcribed using whisper model: large-v2 All right, good afternoon, everyone. I'm going to go ahead and get started. 3.84s - 8.96s | So I'm here today to talk to you about small satellite modeling and defender software. 8.96s - 14.80s | My name is Dr. Kyle Merbach, and I'm from the University of Alabama in Huntsville Center for 14.80s - 20.08s | Cybersecurity Research and Education. And this project has been in partnership with the U.S. 20.08s - 24.16s | Army Space and Missile Defense Command, and we're excited to share with you what we've been doing 24.16s - 32.40s | the last few years. So a little bit about myself. I'm a principal research engineer at UAH CCRE. 32.40s - 38.88s | I've been there since 2020. I've been a PI and SME on various research projects at a variety 38.88s - 44.48s | of different topics, ranging from malware analysis, tool development, reverse engineering, 44.48s - 51.04s | and vulnerability analysis. That can be in various topic areas, including satellites. 51.76s - 55.76s | I have nine years of prior experience as a cybersecurity researcher and software 55.76s - 61.52s | reverse engineer. I got my Ph.D. in cyber operations at Dakota State University, 61.52s - 67.28s | and my B.S. and M.S. in computer security at Rochester Institute of Technology. I'm the 67.28s - 73.68s | technical director of the space testing and resiliency simulation team at CCRE, where 74.48s - 80.64s | we did a satellite project. So a little bit about the project itself. It's been going on 80.64s - 87.52s | since fall of 2019. We have had a wide variety of student teams coming in and out throughout the 87.52s - 95.20s | last five years. So as they come in, they work on various parts of the project, and then they may 95.20s - 101.12s | either move on to other projects or they graduate, and then we hire on additional students and try


chunk_text:
and SME on various research projects at a variety 38.88s - 44.48s | of different topics, ranging from malware analysis, tool development, reverse engineering, 44.48s - 51.04s | and vulnerability analysis. That can be in various topic areas, including satellites. 51.76s - 55.76s | I have nine years of prior experience as a cybersecurity researcher and software 55.76s - 61.52s | reverse engineer. I got my Ph.D. in cyber operations at Dakota State University, 61.52s - 67.28s | and my B.S. and M.S. in computer security at Rochester Institute of Technology. I'm the 67.28s - 73.68s | technical director of the space testing and resiliency simulation team at CCRE, where 74.48s - 80.64s | we did a satellite project. So a little bit about the project itself. It's been going on 80.64s - 87.52s | since fall of 2019. We have had a wide variety of student teams coming in and out throughout the 87.52s - 95.20s | last five years. So as they come in, they work on various parts of the project, and then they may 95.20s - 101.12s | either move on to other projects or they graduate, and then we hire on additional students and try 101.12s - 107.36s | to continue the work. So as a director, I've been the continuee throughout that to make sure that 107.36s - 114.56s | we continue to make meaningful progress. Here is a brief list of major project milestones that we've 114.56s - 120.80s | achieved over the last five years, and we'll go over each of most of these in the slides coming 120.80s - 128.16s | ahead. We'll talk about how we identified and acquired hardware and software to configure 128.16s - 134.00s | a small satellite testbed to model a real-world functioning satellite system, 134.96s - 141.20s | how we try to test for vulnerabilities and create different test scenarios, and come 141.20s - 146.16s | up with ways to harden satellite systems to protect from cybersecurity threats. 147.36s - 152.32s | We developed Small Satellite Defender, which is a software. We'll talk more about that later. 153.04s - 160.64s | And we were able to go to the U.S. Cyber Command Cyber Recon Symposium in Dreamport, Maryland, 161.28s - 168.40s | and we also were able to experiment this Small Satellite Defender software on an actual satellite 168.40s - 174.80s | orbiting Earth with


chunk_text:
101.12s - 107.36s | to continue the work. So as a director, I've been the continuee throughout that to make sure that 107.36s - 114.56s | we continue to make meaningful progress. Here is a brief list of major project milestones that we've 114.56s - 120.80s | achieved over the last five years, and we'll go over each of most of these in the slides coming 120.80s - 128.16s | ahead. We'll talk about how we identified and acquired hardware and software to configure 128.16s - 134.00s | a small satellite testbed to model a real-world functioning satellite system, 134.96s - 141.20s | how we try to test for vulnerabilities and create different test scenarios, and come 141.20s - 146.16s | up with ways to harden satellite systems to protect from cybersecurity threats. 147.36s - 152.32s | We developed Small Satellite Defender, which is a software. We'll talk more about that later. 153.04s - 160.64s | And we were able to go to the U.S. Cyber Command Cyber Recon Symposium in Dreamport, Maryland, 161.28s - 168.40s | and we also were able to experiment this Small Satellite Defender software on an actual satellite 168.40s - 174.80s | orbiting Earth with Lockheed Martin, and we are currently working on assembling and experimenting 174.80s - 182.16s | with TubeSat kits, and we're hoping to continue our efforts there with Small Satellite Defender 182.16s - 196.82s | on those TubeSats. So the purpose of this research project was to come up with a 198.42s - 205.38s | testbed of a small satellite system to be able to perform different research experiments to inform 205.38s - 212.02s | how to better defend satellites from potential cybersecurity threats. As you may know, 212.82s - 219.70s | launching satellites to space has become much more affordable these days with SpaceX launching 219.70s - 223.38s | constantly all these rockets, which is great for our industry. We're able to 223.94s - 232.26s | affordably launch a satellite, and that leads to the need of being able to ensure we harden 232.26s - 238.98s | those satellites from potential threats. So in this testbed that we were developing, 238.98s - 246.26s | we wanted to make it as real-world as possible. So our goal was to be able to operate 246.26s - 252.66s | a payload of some sort. It could be either imagery, GPS, or communications, and


chunk_text:
Lockheed Martin, and we are currently working on assembling and experimenting 174.80s - 182.16s | with TubeSat kits, and we're hoping to continue our efforts there with Small Satellite Defender 182.16s - 196.82s | on those TubeSats. So the purpose of this research project was to come up with a 198.42s - 205.38s | testbed of a small satellite system to be able to perform different research experiments to inform 205.38s - 212.02s | how to better defend satellites from potential cybersecurity threats. As you may know, 212.82s - 219.70s | launching satellites to space has become much more affordable these days with SpaceX launching 219.70s - 223.38s | constantly all these rockets, which is great for our industry. We're able to 223.94s - 232.26s | affordably launch a satellite, and that leads to the need of being able to ensure we harden 232.26s - 238.98s | those satellites from potential threats. So in this testbed that we were developing, 238.98s - 246.26s | we wanted to make it as real-world as possible. So our goal was to be able to operate 246.26s - 252.66s | a payload of some sort. It could be either imagery, GPS, or communications, and then to 252.66s - 258.98s | have the underlying critical infrastructure to support ground station communication, orbit 258.98s - 265.94s | movement and positioning, as well as data processing and battery charging and pointing 265.94s - 271.38s | towards the solar panels to get that charge. And by developing that, we would be able to 271.38s - 279.22s | replicate a real-world system. So in the very beginning of this project, we spent time 279.22s - 286.34s | researching, trying to figure out what the best avenue was to build this testbed. And we were 286.34s - 293.70s | looking at the Raspberry Pi 4 as well as the Pi3 version 4 as well. We found that the Raspberry 293.70s - 300.42s | Pis were relatively easy to acquire, obviously everyone's badges. It's fairly inexpensive, 301.62s - 307.70s | simple to modify payloads and add payloads. There's a lot of pre-existing libraries already 307.70s - 313.14s | configured. There's been a lot of research done out there in the community. Whereas the PiQ boards, 313.78s - 321.86s | they were developed to be nanosatellites, and we found it very difficult to be able to write to 321.86s - 328.18s |


chunk_text:
then to 252.66s - 258.98s | have the underlying critical infrastructure to support ground station communication, orbit 258.98s - 265.94s | movement and positioning, as well as data processing and battery charging and pointing 265.94s - 271.38s | towards the solar panels to get that charge. And by developing that, we would be able to 271.38s - 279.22s | replicate a real-world system. So in the very beginning of this project, we spent time 279.22s - 286.34s | researching, trying to figure out what the best avenue was to build this testbed. And we were 286.34s - 293.70s | looking at the Raspberry Pi 4 as well as the Pi3 version 4 as well. We found that the Raspberry 293.70s - 300.42s | Pis were relatively easy to acquire, obviously everyone's badges. It's fairly inexpensive, 301.62s - 307.70s | simple to modify payloads and add payloads. There's a lot of pre-existing libraries already 307.70s - 313.14s | configured. There's been a lot of research done out there in the community. Whereas the PiQ boards, 313.78s - 321.86s | they were developed to be nanosatellites, and we found it very difficult to be able to write to 321.86s - 328.18s | the boards. There was very little documentation available, and unfortunately when we acquired 328.18s - 335.22s | that hardware, it was end of support. So we shifted back to the Raspberry Pi platform moving forward. 337.14s - 342.34s | So this is an illustration of what we were trying to achieve with these Raspberry Pis. 342.34s - 348.34s | We would have a satellite and the ground station. With the satellite, we have a payload, which in 348.34s - 353.62s | this case we went with a camera where we can take pictures. We have antennas on both the satellite 353.62s - 359.54s | and ground station. We're able to communicate to each other, send commands. And as you can see, 359.54s - 364.74s | we had an encasement and some ports to be able to connect with it with a laptop, be able to 365.94s - 374.10s | upload code, configure things, and so forth. And this is the real world image of what we created 374.10s - 380.10s | in the lab. We were able to replicate that diagram. And essentially the camera, the radios, 380.10s - 386.58s | and everything we ordered and put together pre-loaded libraries.


chunk_text:
the boards. There was very little documentation available, and unfortunately when we acquired 328.18s - 335.22s | that hardware, it was end of support. So we shifted back to the Raspberry Pi platform moving forward. 337.14s - 342.34s | So this is an illustration of what we were trying to achieve with these Raspberry Pis. 342.34s - 348.34s | We would have a satellite and the ground station. With the satellite, we have a payload, which in 348.34s - 353.62s | this case we went with a camera where we can take pictures. We have antennas on both the satellite 353.62s - 359.54s | and ground station. We're able to communicate to each other, send commands. And as you can see, 359.54s - 364.74s | we had an encasement and some ports to be able to connect with it with a laptop, be able to 365.94s - 374.10s | upload code, configure things, and so forth. And this is the real world image of what we created 374.10s - 380.10s | in the lab. We were able to replicate that diagram. And essentially the camera, the radios, 380.10s - 386.58s | and everything we ordered and put together pre-loaded libraries. So for the camera, we went 386.58s - 394.18s | with the Aokin Raspberry Pi cameras capable of taking very low quality images, but just for the 394.18s - 399.94s | purposes of showing that you can use payloads with the system we were setting up. And that uses 399.94s - 412.74s | pre-existing Raspberry Pi libraries. We also used the RFM69ACW radios that attach directly to the 412.74s - 420.02s | GPIO header on the Raspberry Pis that function at two different frequencies. It had a decent range. 420.02s - 424.66s | We were able to go as far as 100 meters without any problems. We never had any need to go any 424.66s - 432.29s | further than that for the purposes of this project. And for the communication with these 432.29s - 439.49s | radios that we had in place, we set up a TCP style communication between the ground station and 439.49s - 445.97s | satellite. The satellite would interface with cesium. We'll talk more about that later. And then 445.97s - 451.97s | we would simulate this orbit window. And then when it would be in the communication window, 452.85s - 459.25s | the satellite


chunk_text:
So for the camera, we went 386.58s - 394.18s | with the Aokin Raspberry Pi cameras capable of taking very low quality images, but just for the 394.18s - 399.94s | purposes of showing that you can use payloads with the system we were setting up. And that uses 399.94s - 412.74s | pre-existing Raspberry Pi libraries. We also used the RFM69ACW radios that attach directly to the 412.74s - 420.02s | GPIO header on the Raspberry Pis that function at two different frequencies. It had a decent range. 420.02s - 424.66s | We were able to go as far as 100 meters without any problems. We never had any need to go any 424.66s - 432.29s | further than that for the purposes of this project. And for the communication with these 432.29s - 439.49s | radios that we had in place, we set up a TCP style communication between the ground station and 439.49s - 445.97s | satellite. The satellite would interface with cesium. We'll talk more about that later. And then 445.97s - 451.97s | we would simulate this orbit window. And then when it would be in the communication window, 452.85s - 459.25s | the satellite would send data to the ground station based on its previous tasking. And then 459.25s - 465.09s | they would listen for additional tasking from the ground station. And we would control different 465.09s - 477.12s | states for the satellite with the first byte of the data transmitted as an identifier. So in this 477.12s - 484.32s | custom communication setup that we had, we had various bytes. We would let the satellite know 484.32s - 488.96s | this is the end of the stream, or this is the start of a specific stream for pictures, 488.96s - 499.79s | or it was a task, or it was for a configuration on the satellite. We also acquired the SenseHat 499.79s - 506.03s | add-on board. This was originally developed for the International Space Station by the Raspberry 506.03s - 513.71s | Pi Foundation. This allowed us to have an array of sensors with a variety of different temperature 513.71s - 520.11s | communities. And the big thing for us was the gyroscope being able to position the satellite 520.11s - 526.11s | to be able to figure out which way it's facing, which is really important for yaw and


chunk_text:
would send data to the ground station based on its previous tasking. And then 459.25s - 465.09s | they would listen for additional tasking from the ground station. And we would control different 465.09s - 477.12s | states for the satellite with the first byte of the data transmitted as an identifier. So in this 477.12s - 484.32s | custom communication setup that we had, we had various bytes. We would let the satellite know 484.32s - 488.96s | this is the end of the stream, or this is the start of a specific stream for pictures, 488.96s - 499.79s | or it was a task, or it was for a configuration on the satellite. We also acquired the SenseHat 499.79s - 506.03s | add-on board. This was originally developed for the International Space Station by the Raspberry 506.03s - 513.71s | Pi Foundation. This allowed us to have an array of sensors with a variety of different temperature 513.71s - 520.11s | communities. And the big thing for us was the gyroscope being able to position the satellite 520.11s - 526.11s | to be able to figure out which way it's facing, which is really important for yaw and axis, 526.11s - 531.79s | making sure the satellite doesn't spin out, and make sure that it's orienting the solar panels 531.79s - 538.51s | in the right direction to get the energy to charge the batteries. With this add-on board, 538.51s - 542.11s | there are a wide variety of different functions available with those sensors. 543.63s - 548.59s | We would want to position the payload in the right direction to be able to take pictures 548.59s - 554.51s | of Earth, and we would also want to protect the camera from not facing the sun, which can burn 554.51s - 561.87s | the image sensor. We would orient the solar arrays using those positions, and we were able to measure 561.87s - 567.63s | the orbit speed and g-force using the accelerometer built on the SenseHat board. 572.24s - 578.24s | With all of these different components that we put together, we also had to keep track of 579.12s - 583.92s | the satellite orbiting Earth. Obviously, it's in a lab just sitting on a table, 583.92s - 590.16s | so we had to simulate the actual orbital movement. So we tried a variety of different 590.16s


chunk_text:
axis, 526.11s - 531.79s | making sure the satellite doesn't spin out, and make sure that it's orienting the solar panels 531.79s - 538.51s | in the right direction to get the energy to charge the batteries. With this add-on board, 538.51s - 542.11s | there are a wide variety of different functions available with those sensors. 543.63s - 548.59s | We would want to position the payload in the right direction to be able to take pictures 548.59s - 554.51s | of Earth, and we would also want to protect the camera from not facing the sun, which can burn 554.51s - 561.87s | the image sensor. We would orient the solar arrays using those positions, and we were able to measure 561.87s - 567.63s | the orbit speed and g-force using the accelerometer built on the SenseHat board. 572.24s - 578.24s | With all of these different components that we put together, we also had to keep track of 579.12s - 583.92s | the satellite orbiting Earth. Obviously, it's in a lab just sitting on a table, 583.92s - 590.16s | so we had to simulate the actual orbital movement. So we tried a variety of different 590.16s - 598.64s | open-source software. The one that worked the best with our setup was Cesium Orbit software, 598.64s - 604.80s | and as you can see in the pictures, we were able to configure two different views. You can see from 604.80s - 612.48s | the satellite view to Earth, and you can see as it moves around the world, and whenever it would 612.48s - 619.20s | come into a range of this ground station, you'll be able to see the pre-configured GPS location that 619.20s - 624.32s | we would choose to be able to simulate that communication window opening and closing. 625.20s - 630.80s | And as that orbit would occur, that's when we would transmit messages back up and down from 630.80s - 641.79s | the ground station to the satellite. Once we had this test bed working well and functioning 641.79s - 648.27s | the way we wanted it to, we started to move towards coming up with different task scenarios 648.27s - 653.87s | and trying to come up or find different vulnerabilities that may be of interest 653.87s - 658.67s | to try and make sure that a satellite that a researcher


