- I'm Natalie Russell and I am an accessibility program manager at UserTesting. I've been with the company for about a year and four months now. And I've been in my current role for around six months. Previous to that, I was a quality engineer for around 12 years. I'm hugely passionate about making the world a more accessible place, particularly in the digital space. This is partly because my mom is disabled. She has several autoimmune diseases that have left her with a lot of chronic and debilitating pain. So she struggles a lot with technology due to the dexterity of her hands and her eyesight. So this is something that is really close to my heart. So I wanted to come today, talk to you about all the amazing things that we've been doing at Deque, and hopefully give you some advice and some tips on how to start your accessibility journeys. Okay, so this is our agenda for today. I'm gonna talk a bit about who our UserTesting, our values, our product, and who we work with. I'm gonna talk about our journey from reactive to a proactive model, including the process of selecting the right accessibility partner and what qualities you should look for. How to determine the scope for your audit to make the most impactful changes for your users. And what areas you should focus on when you're starting an accessibility journey. I'll then discuss like my three pronged approach for what we've done at UserTesting and share my tips for successful triaging and tracking the remediation efforts against your audits. I'll then talk about continuous testing and how we're developing strategies for sustainable organizational accessibility and our culture change. So what's gonna happen in the future at UserTesting. I'm just gonna have a sip of my water. Thank you. Okay, so first off, I'm gonna play you a little video about UserTesting and what we do here. I'm just going to turn on the captions for this as well. Let me know if you can hear this.

- [Narrator] Finally, there's an easy way to watch people use your app, product, or service. UserTesting lets you see the experience through the customer's eyes by watching what they do while hearing why they do it. For example.

- Oh, this is great. I love this. I'm gonna buy two of them. So I have to hit the plus sign, but it's so close to the chat. Let's see. Oh and I must have pressed the chat.

- [Woman] We just got this body wash. I'm trying to figure out how to get the pump up so the pump will work and I've twisted a few times counterclockwise and nothing is happening.

- It says quantity one. But here it says two for 14 79. I'm not sure if I'm buying one or two. Now it says three at 14 79. But here it seems to be saying three plus three free. I don't know if I'm ordering one, two, three, or six.

- [Narrator] UserTesting helps you avoid problems like these with 80% of feedback coming back in just a few hours. Here's how it works. First, choose your target audience. You can select from a global network of participants. You specify exactly type of person you're looking for and we find them for you. Or you can even use your own customers. Next, select what to test such as early prototypes, websites, mobile apps, internet of things, competitors, omnichannel, so you can see your customers end to end journey, marketing content, like so social media or blog posts, or real world experiences.

- [Man] Wow, I like the feel, feels really solid.

- [Narrator] You can test pretty much anything digital or physical. Next, create tasks you want the participants to do like buy a plant online. You can even start from a template that has tasks and questions written by experts, such as testing your app against a competitor. During the test, participants record their screen and speak their thoughts out loud while performing tasks you give them.

- [Woman 2] I'm glad it arrived the right direction, facing up. It looks well protected.

- [Narrator] Or you can have one-on-one online interviews with participants. You can get instant feedback from your customers as they interact with your products. You can make video clips and use those clips to create a highlight reel. We also offer metrics, transcripts, and sentiment analysis using machine learning that automatically calls attention to the biggest issues to help you zero in on the problems that participants encounter. Our professional services team can do the research for you or recommend the best practices for empowering all your teams to empathize with your customers. More than half of the world's top 100 brands rely on UserTesting. Join those leaders and you'll dramatically improve your customer's experience. UserTesting, real human insight.

- So I just wanted to share that to give you an oversight of what UserTesting does and how we are interacting with our customers and how we have our panelists and people giving that real human insight. Sorry, I'm gonna take you through, I've mainly wanted to share this. These are our five core values at UserTesting as you can see they are get better, drive results, customers first, be kind, and keep it simple. So I'm just gonna talk briefly about these. Get better. We really embrace like a continuous learning strategy at UserTesting so that we learn and evolve and grow as a team. We drive results by striving for excellence in all that we do to contribute to our shared success. And we're focused on results and working with a sense of ownership and urgency. We put all our customers first. Our most important job is to help our customers be successful, right? So we're very detail oriented and we earn trust through our expertise, our integrity, and our reliability. Be kind, this is my favorite value and it's probably the reason I joined UserTesting as well. This makes us stronger. We are a humble company. We're authentic. And we treat each other with the upmost respect. We are trusting and we always assume positive intent. And then keep it simple. So this doesn't always mean like easy in that sense. We're intentional and we're focused and we believe in quality over quantity. So we reduce ideas and solutions to their most seamless and elegant options. So yeah, so these are like one of the many reasons that I joined UT. I love working here. I love the people. And I love the values. I love our company culture and all the different employer resource groups that we have, like Access Abled and UT Cares where we raise lots of awareness and donations for charities. So we are really recognized for all our efforts as employees and I love the product and the vision of delivering through human insight. So this is just an infographic showing some of the customers and the companies that we work with. We help to bridge their empathy gap for them so that they can truly understand their customers. And I've just got a few steps for you here. Like it said on the video, over half of the world's top brands are actually powered by UserTesting. We have 650 plus employees globally and we're growing all the time. We actually went public last year as a company. We've even written a book on user testing. So we are leaders in our industry here and we actually had 20 million minutes of videos captured in 2020. So lots of real insight there. So in short, our customers love us and we love our jobs. And as a result, we have like a really fantastic opportunity of course, to make the world a more accessible place through the customers that we serve and the reach that they have. And that helps them to enable their customers and bridge that accessibility, empathy gap with theirs. So just to be clear, I'm not talking obviously about running accessibility checks or auditing, we'll leave that to the experts like Deque, but really more about that accessibility experience. Okay, so I'm gonna talk a bit about our journey so far. So this really started with the vision from our Accessibility Guild and within the company to improve our accessibility experience in compliance of our platform. So that all disabilities, both visible and non visible, are reasonably accommodated for. So to do that, we adopted the web content accessibility guidelines up to level AA for creating more accessible digital content. In 2020, the Accessibility Guild and the security team determined that we needed to complete a full accessibility audit. And we were already in a really great position at this point. Our process had a lot of backing internally from our CTO, Kaj van de Loo and our CEO, Andy MacMillan. And I can't stress how important this part is to anyone starting this journey. And I think this part of the process is becoming a lot easier as the need for accessible solutions, heightened even more so by COVID, has meant that there's a lot bigger focus on this. And at the end, of the accessibility is a human right. It's the right thing to do. And as I said, when you saw it fully aligns with our values and our desire to improve our digital experience. Also the fact that it's a $10.3 billion eCommerce market is obviously a very compelling reason as well. So yeah, so in August, 2020, we started evaluating accessibility vendors. Some which we'd had internally recommended and some which we'd found just using Google. We evaluated six in total. Three of which we weren't able to work with because they weren't able to assess both our mobile and our desktop platforms, which was basically a hard requirement for us through the the scope. And two of the vendors that we actually spoke to recommended Deque and said that we should speak to them. So that's exactly what we did. And we partnered with Deque in November, 2020. So the reason we partnered with Deque, obviously being industry leaders in this space, sorry, I'm just gonna change my slide. Sorry, two seconds. Sorry. So yeah, we selected Deque as our vendor, obviously as their own industry leader and they came highly recommended from other companies that we spoke to. They were the only company that were able to facilitate all our needs and be able to scan all of our platform. They employed more certified accessibility experts than anyone else in the world. And they could give us the tools, these industry leading tools to be able to audit, monitor our sites and our apps for accessibility issues. So it's really important to work with them. Sorry. Okay, so partnered with Deque. So then we spent around three to four weeks defining the scope for our audit of our core functionality across our platforms. Defining the scope meant obviously a lot of collaboration across our internal delivery teams to ensure that we had the right level of coverage. You don't need to scan and you shouldn't scan everything. So picking those core processes throughout your platform, your internal at development teams, your business leaders, and your analytics will give you insight into what you should be covering as a baseline for your customers. I actually joined the company in November, 2020. So I joined the Accessibility Guild and I was so thrilled that we were already partnered with Deque. I'd use the testing tools, the built-in browser testing tools, throughout my career as a QE, so I knew that this was a great partnership and the right way to go. And I've always been passionate about accessibility. So I was thrilled to see that it was already part of this company. So in January of 2021, we computed the first audit and these results as a say, both our platforms and also our marketing website. And the scope was focused as I say, on our core journeys throughout our platform, so that we could understand what our current conformance level was against . So then we spent, so by the time we got to Q2, obviously, we were starting to review and address our audit results. I'm just gonna have a sip of water. So triage in and a sign in the works that took a few months as there wasn't a dedicated team at the start of this for this work at that time. So I'm a Jira admin and I'm a big Daya nerd so seeing all this, I wanted to dive in and help assigning the Jira tickets. We use Jira as our ticket managing system, to make sure that the tickets had the right level of detail, to help organize those audit results, and ensure we were assigning the correct issues to the correct teams. Again, this required a massive collaboration across our product design and engineering department, so that we could identify and prioritize those highest severity issues first. What was a blocker to our customers, basically? We leaned heavily on our internal tools like Jira and Slack to ensure we have the right level of detail and people understood the issues that needed to be resolved. And this process was flawed initially due to the amount of time that it took to get all the tickets into place and find the right people to be working on them. We have around, I think it's 21 different squads now, around 175 people in that department. So it was a lot of work to get us all together. So between March and May, we focused on resolving those critical issues on our core journeys and focused a lot of effort, and this is a really good tip, if you don't have one already, having a design system to have reusable components across your platform. This is where we saw the most increase in our accessibility levels. We were able to focus on that, which really helps from both the UX and UI design perspective, but also from a code repository perspective. It also helped us resolve a ton of common issues like color contrast, keyboard navigation, and to have consistent roles and programmatic labels. It also helped us reduce the level of duplication found across the board, 'cause we have a design system and we were also finding issues on the page. So it meant we could put these into this correct team and have them resolved globally across the platform. And while I was doing this, I just wanted to show a picture of the "Agile Accessibility Handbook" that was written by Dylan Barrell at Deque. And this literally has been my Bible. This has helped me so much throughout the journey, giving me practical tips on how to start this journey and to take the incremental steps needed to be successful within your journey. Oh, so sorry. So as part of Q2, just in the middle there, we also re-audited the same scope at that time so that we could measure our progress and updated our voluntary product accessibility templates. I mean, it showed that we've made some great progress to work at high compliance. And we were starting to implement a great culture shift within our processes. We'd run a week long training session in May for Global Accessibility Awareness Day, for the department with presentations and examples on how to improve our code and practices. And since then, and to date, we've been auditing all our new work. So everything that's been in beater or been released, and this is so hard to, sorry, this has started to help us see how much the work that we're doing is actually impacting our compliance results and our experience overall. So to keep the focus and the momentum going, I give weekly updates on Slack and shout outs all the teams that are doing such fantastic work, also at our department meetings to thank all the designers and engineers that have been working to resolve these issues. And also sharing empathy moments on the platform where we've had a particular issue and we've resolved it and seeing that how great that work is. So this was really to keep like inclusivity, the requirement of accessibility, top of mind for everyone. And then in Q4, we partnered with Deque's accessibility program office to start our own program office. And I'm gonna talk about that a bit more shortly. So excuse me, I'm just gonna skip these slides. Oh, I've clicked the wrong button. Sorry, two seconds. Okay, so as you can tell, we were very much in a reactive process at this point. We'd had our first accessibility audit completed by accessibility specialists and actually this can be quite scary. It's kind of like taking a test that you haven't really studied for and then the messaging to your teams has to be done correctly. So it can be quite overwhelming having tickets added to your backlog that you were not expecting or that you don't feel potentially you have the skill set to be able to resolve. And my tip here is not to add SLAs or expected resolution times initially. This only puts unnecessary pressure on your teams and it can be quite demotivating. You have to do the education and the skilling grace gradually as you go through this. So step two of a reactive process is to prioritize all your accessibility issues, depending on what's important to you as well. So severity is how we went to look at our platform and looked at how the traffic impacted that and other criteria that was important to us. It's the best way to approach addressing accessibility issues on existing sites and applications internally. Step three, fix those accessibility issues. Get dedicated time in your sprints and your roadmaps to work on resolving existing issues. This means that it's prioritized as part of your features and your other business values to ensure that you are really delivering that quality. And then lastly, perform validation testing to ensure that the accessibility fixes are actually working. And you can do this both internally and externally. So we've been using a mix of both. So like I say, obviously at this point, we're in a very reactive model and Deque really have been helping us move from this model to a more proactive approach. So I love this little picture just because it's like building blocks and turning that over from being a reactive process into proactive and moving forward. So just gonna sip my water. Try not to choke. Excuse me. Sorry. So why should we do this, you might ask. So the cost of fixing accessibility defects and discovering them in your after release or in your production is about 30 times more than if you catch them in the design and architectural phases. So this is a graph from IBM that Neil from Deque kindly shared with me, and it just shows how implementing a shift left process means that you're going to find a lot of your issues a lot earlier, and you're gonna be able to produce a more accessible solution. This is just another example showing you how traditionally we audit and check things after we've released things. So that takes a lot of large investment and that shifting left making smaller, incremental investments throughout your process with your backlog, your design, your coding, and your testing means that you're gonna see results a lot quicker and in a lot more structured way. Sorry. Okay, so moving forward to what we need to do to be proactive. So step one, you need to train everyone in accessibility concepts and techniques. So a proactive accessibility plan not only puts your organization, obviously, at a lower risk of receiving complaints about accessibility, but it also makes the entire accessibility process more efficient, more cost effective, and as non disruptive as possible. So training everyone is really important. I'm gonna talk about what we did around this in a second. Step two, get any of your new design wire frames reviewed for accessibility optimization and potential issue identification. So this is a crucial step and thankfully Deque provide a service for this. I'm gonna talk about that again in a second. Step three, equip your development teams with the right tools to integrate accessibility testing into all stages of your development process. And thankfully again, Deque have a lot of great tools around this that can help you really embed this into your process. And step four, establish internal policies and processes to ensure the accessibility of all your content. And that's everything from your digital products, your content, your third party tools that you provide to your users and your employees as well. Okay, so this is my three-pronged approach. So using Dylan's amazing book and thinking about what we needed to do to do this shift left. I started in this sort of incremental way. So first off we were already completing audits on a regular basis, on a quarterly basis, and having comprehensive scans across our mobile and desktop platforms. We then continue resolution and testing of those audit issues prioritized by their severity. We then started to improve our processes. Obviously, as I said, we're engaged with Deque to help us build our accessibility program. And we've started to shift left in our processes and embed this really into our life cycle. And then education. So we've created lots of learning paths. We've had tailored training for the entire department, process and tools, workshops, and started to build an accessibility knowledge base so that people could get information easily and understand what the right fixes are to implement and at what time. So, I'm kinda gonna talk about this backwards in a sense. So I'm gonna talk about what we did for education first, 'cause as you remember from the step one of the proactive plan, that was the main thing that we needed to do. So with the help of many people in our learning and development committee, we ran a three day training exercise for the product design and engineering department. So this was run over three Fridays in November last year and we combined empathy training, learning activities, and a whole day of action. I think everyone will probably laugh in UT. I actually called these all the athons. So we had a innovationathon, an automationathon, a hackathon, and a documentathon. So that we could start looking at all the areas that we needed to and being quite a big department with lots of different disciplines within that, we really wanted to put a program on that everyone at any level of their journey in accessibility had access to and were you easily able to take this. So day one, we kicked off with a big kickoff presentation to talk about what the tag was, what was accessibility, and give everyone that baseline understanding and to align everyone the vision for those days. And what we hope to achieve. This was a global event for 150 people. And the event, the kickoff, we actually ran twice to cover both of the time zones, making sure that everyone was included and coordinated and aligned. So we then challenged everyone to complete six empathy lab challenges that were designed to achieve three things. So gain experience and awareness of different impairments and disabilities, learn about the assistive technology that helps improve the quality of life, and learn more about our platform and the experience with assistive technologies and simulations. So we did six of these, There was one for blindness and low vision, color blindness, physical and mobility impairments, hearing impairments, cognitive impairments, and reading impairments. And I'm very happy to say that they were so successful that we've actually just rolled them out across the whole company. So it's given everyone an opportunity to build their empathy and their understanding around this. Yeah and then on day one, we actually had Tim Harshbarger from Deque, who is blind, come in and talk to us about his experience and his views on digital accessibility. And I can't tell you how powerful that was for everyone. I learned so much just from him coming in and talking to us about his experiences and views. So it was really inspiring and educational. And I highly recommend that you do this. On day two, we combined a number of A-sync learning opportunities using Deque University. and . And this allowed us to offer like tailored training at all levels and disciplines across our department. And being a global company meant that people could take these in their own time as well. So just being kind about people's times. And then we have the lovely Preety Kumar, who is the CEO of Deque, come in and present to our teams about Deque and their vision. And we had a full demo of the axe dev pro tools, which we had already started to implement into our process, but this obviously led to an increase in an uptake in people that wanted the licenses, particularly for our developers who really hadn't seen the full benefit of that before. So again, a really good tip to bring them in to have that discussion. We ran team workshops to complete basic page audits, covering what we can do automatically like running that axe dev tools and interpreting those results. And then what we need to test manually, like zooming in on pages, keyboard navigation, and screen reader checks. So these again were really successful as the team's got hands on and practical application of the tools available and understood the basic level of coverage that we needed to be accessible. Excuse me. So, yeah. So and then in day three, we did all the athons. So as I said, hackathon, automationathon, documentathon, and an innovationathon. So this was partly as I said, to cover all the disciplines within the department so that everyone could get involved and we could remediate some of our issues, discuss and define what our processes should look like, and what we needed to do to implement those and improve our processes. And then what opportunities we have for innovation, like implementing specific tools and features for accessibility. And across the days we had social activities to build empathy with our teams. We played lots of games, had lots of quizzes, and awarded prizes and had a budget for food. So it was really good fun. And these are just some of the quotes from the department or from people that joined us. So I'll just read these out. "The days were fun packed, and also easy to drop in at your own schedule. I really liked there being a focus to these days." "I really enjoyed the mixture of collaboration, learning, and games. It was fun." "It was engaging educational, inspirational, and a great experience that I want more of." So I think it's really important, obviously, in an agile framework to measure how successful even things like your events and your empathy, or your ongoing empathy campaign, so that you can understand, you know, are you getting the right level of learning out there? Are people finding that content accessible? And what more can you be doing around that to increase that awareness? So that moves us on step two of our proactive plan. So that is to get any new wire frames reviewed for accessibility. So we did two things here cause you can get, obviously it's really important, to start that shift left. So we wanted to start internally first by implementing a really solid checklist and to start embedding that into our design process. And we now have all our major new designs going through an accessibility design annotation process, which is provided by Deque. And this is gonna help us improve and think about all the roles, names, and interactions required. And the benefits of this is that it provides input to both the developers and QEs when implementing a solution and writing automated tests for it. The engineers can use this information to choose like the correct semantic elements, the markup, and then use them to validate their choices. And as most of us will know, having the correct semantic HTML, and elements being used for their correct purpose, it means your pages are more accessible right out of the box. Content is more readable by sighted users as well, but it'll also be usable by assistive technologies like screen readers. It also helps to identify areas where the designs might be overly complicated and where elements might have multiple controls or interactions. So an example of that is like we've got a menu item that opens a link and a sub menu, for example, and that's fine for a mouse user, but it would block a screen reader and a keyboard only user. So this is what we've been doing to start our shift left. So step three is to equip your development teams with tools to integrate accessibility, testing at all stages of your development process. So from the first audit, we'd already started using axe dev pro tools and implementing a quality checklist to help us guide our manual testing. We embedded those pro tools into our development and testing sorry, practices. So QEs run manual testing for like screen readers, zoom, and keyboard navigation. And the scans are run by developers when they're building features so that they can find any programmatic errors and resolve them before the code is actually implemented. So the few things that you can do additionally around here to make sure that people are supported. So for example, we have a GitHub group specifically for reviewing accessibility fixes. This has helped us like keep consistent across the solutions that we're implementing and making sure that we're fully resolving the issue that has been raised. And we use our Slack channels, obviously, a lot to help us internally. And we even have a Slack channel with Deque as well, which is super helpful and just means we can get quick advice straight away. We use intelligent guided testing that is available and increases our coverage massively. It can increase your coverage up to 84%. If you're not using this, you should be. And at least one member of every squad that we have has a pro license and everyone in our QE and mobile teams has one. And as I say, this has helped us like identify and raise more issues automatically and validate that our audit issues have been resolved. Adding automated checks, obviously, means that we are covering approximately, I think it's 30%, of our programmatic issues. These are really fairly easy to implement and we use Cyprus as our automated testing and our architect actually helped us devise an interim approach for this because, so the goal for this in the end obviously is to have all of your PRs gated, right? So you're not releasing any code with new accessibility issues in. So we're covering it at those early stages as we come through the development pipeline and then we're also covering it from an automated approach. So we've gone for that interim approach at the moment, so that we're recording the issues that we've got per page. And then we're able to see as they're resolved and as things come through the pipeline that that's been resolved, and then we can close that issue. Yeah, so just ensuring, like I say, that we're not introducing any new one. And I say, we went for this approach with the end of all, to implement this fully into our CICD, continuous integration pipeline and to block any changes that have accessibility issues. So I highly recommend that you do this 'cause you get a lot of things for free and it reduces that amount of manual testing that you need to do. Okay, so what's next for us? Okay, so as I said, just earlier on, we are partnered now with Deque's accessibility program office. And this again is a huge collaboration piece across the company. We're in discovery at the moment with Deque. So they're helping us to measure our maturity using the digital accessibility transformation index or the DATI. And that's gonna help us implement a playbook tailored to our specific needs. And that's gonna help us to plan our implementation incrementally. The discovery interviews that we're doing currently are a really crucial part of to this process so that it allows us to have informal question based discussions with everyone in our organization and our department to see how structured we are, like the current methodologies that we're using throughout our solution, delivery, life cycle, and other areas of the company like procurement support, legal, leadership, everyone at the company literally has been involved in this initiative. So for the next, the remainder of this year, we're building out our accessibility program office and our strategies for continuous improvement. We'll be writing our own accessibility policies and embedding accessibility fully into our processes. Now there are 10 elements of an accessibility program. So the interviews have been tailored around each of these areas. As you can see, that's the development life cycle, your testing and validation, how you're implementing your training, your governance and risk management, if that's applicable to the type of work that you do, having a policy, this is the biggest thing that I'm excited for so that you can, you know, say to your customers and say internally, what your standards are and how you are implementing this process, your legal teams, your fiscal teams, your procurement process. Particularly working with third parties, you've integrated them into your platform. It's really important to have processes and policies in place so that when you are working with people, you can ask them to be accessible as well and say what level that you are accepting at. And obviously have that conversation about how you fix issues together, if there's things in the platforms already. Communications obviously is a massive thing, how you communicate this process and your plan to your internal teams and your customers, and then also the support that needs to go on around that. And we've done a lot of training internally in lots of different areas here, particularly in support so that we can make sure that we're enabling our support teams to give them right information. And that they're comfortable with the recommendations that they're making. So, I've finished really early, I realize, but hopefully that's more time for questions and thank you for joining me.

- Awesome. Thank you so much, Natalie, for a great presentation. I mean, it's just really amazing to see a real example of an organization transitioning their accessibility practices and processes from reactive to proactive. I mean, thank you for sharing that experience with us and thank you and the rest of your team for your commitment to accessibility. Appreciate that.

- Thank you. Thank you for your time. We couldn't have done this without you. Your advice has been, and your support has just been amazing. I, yeah, so thankful to everyone at Deque.

- Thank you. Alright, so we've got a little bit less than 10 minutes and we've got lots of questions. So just to remind everyone, please continue to submit your questions and also vote for the questions that have been submitted and we'll go through those questions. So, alright, so the first question I have here is you mentioned empathy lab. So the question is, did you build your own empathy lab experience or did you hire an organization to do it?

- We actually built our own internally. So being the type of platform that we are obviously, we are able to create tests. So we used our internal platform and the tests were really built up with some educational videos and some information, so that you understood the types of disability that you were learning about. And then we had practical examples of how to create things actually within our platform. So it was kind of super meta in that sense because we had people internally taking tests on our platform as customers, but then they were actually, sorry, as contributors, and then they were actually building tests out as a customer using accessible tools like a screen reader, to build a test plan and to launch a test. So it was really interesting to see the results and there's some really great simulators out there as well that you can add to your browsers to be able to see different types of impairments, especially color blindness. I think people really enjoyed that to see the difference and see how important it is to not just use color alone, to show links and information like that, because it can be such a huge blocker when you have that type of an impairment.

- Thank you. Alright, the next question I have is, how did you monitor remediation progress of your dev teams? And did you collect any metrics? And if so, if you can give some example of some of the metrics that you were collecting to monitor progress.

- Yeah, totally. We used a lot of Jira dashboards to be able to do that level of reporting and communicate where things were in the pipeline, what status they were in, and many issues per team we had, and what level of criticality that was. So that was the main thing that we focused in. You know, what area of the platform is this in? So is this something that is in a reusable space? So in our design system library. Is it something that we can do a global solution for? Or is this something that lives particularly on a page with a specific team? So making sure that things were assigned in the right place. And we have domains within our Jira application so you could see exactly where the issues are, where the biggest problem areas were, and it really allows you to drill down and give the teams a lot of advice around what they're doing. The other thing that I've been doing a lot of is a lot of analysis of those results. So really have a look at the areas that you are having repeated issues in so that you can think about what those solutions are. And obviously you're not gonna solve everything straight away. There are things that need to go back to your product teams, to your leadership in that sense, to have the conversations about, how do we meet this accessibility need? How do we grow our platform so that we can reach all of the different types of everyone and gain this real true human insight by enabling everyone on our platform? So, yeah, criticality, where it is in the platform, and making sure you're reporting that as well. So people understand that remediation and obviously given lots of shout outs. You know, we had, I think it was a couple of weeks ago, we had like loads go through the system and just so nice to praise people for all the hard work that they're doing around this.

- Excellent. Alright, thank you. Alright, next question I have is, how do you measure your maturity of an accessibility program?

- We've been doing a number of things and this is why we've partnered with Deque because this was the part I was like, I wanna be really confident that we are growing that and that we are going in the right direction. So that's why we chose to partner more solidly with Deque and their program office so that we could have strategic consultants come in and help us to do that basically, and figure out are we measuring the right things? Are we going in the right direction? But you can look at your audit results. I think that was what was good for with us, you know, specifically doing quarterly audits on all our new features. And that's where really where we've seen the most impact, because you can see the compliance levels going up. You can see the right things are being put into place and that all that hard work that you are doing in these shift left positions is starting to really pay that off. So that's what I track around. Deque also have a wonderful certification program that you can use. You're never gonna solve every issue on every page, but you should be trying to do that. Reduce your overall number of issues per page and focus on your biggest criticality ones first. And then make sure you've got solutions for everything as you go through and that you're covering all the impairments.

- Awesome. Thank you. So speaking of audit and audit results, next question is, what was your biggest process improvement as you assimilated the results of the audit?

- Oh, that's a really good question. I think for me personally, when I first started, you know, loading in the issues into Jira and getting everything organized and into the right teams and the right places, that's the thing I've improved the most because at the start of that it took, especially 'cause I was new to the company as well, right? So I didn't know who the best people were to speak to or who I should go to for advice or who we should assign this to. And there was a bit of a disconnect between, so we've got these audit results and this is the page that this has been found on, but that doesn't tell me which team that belongs to or how that should work. So now what I've been putting in place, and as we've developed the scope for each audit, I get the teams involved from every level from design and engineering and QE, QEs have to shout out my teams 'cause they've been absolutely instrumental in helping me build that scope, so that it's a lot easier for Deque to complete the audits. They're really clear on what areas we need to focus on and what we're looking at in that sense. Yeah. Does that answer the question? Sorry.

- Yes. Thank you. Alright, so along those lines about, you know, getting an audit report and then, you know, start remediating those issues. So the question is, what advice would you give a development team that is struggling to remediate accessibility issues?

- Talk to each other. Make sure you've got good communication and support lines. So we, like I said, we have an accessibility GitHub group. We have internal channels specifically for dev and QE. And then if we can't figure out the solution or the answers between us, and you have varying levels of experience throughout your organization, some people will know a lot about this. Some people people know very little. So get each other talking to each other, get training put in for your development, and your QE teams and your design teams specifically because they need to talk to each other as this process shifts left. So making sure that you're putting the right things in place, so communication, make sure you've got that. And a knowledge base like having, we use Confluence. So having accessibility knowledge base and being able to give people quick access to like, okay, so you need to implement old text. What do you need to do around that? You need to do, you know, a focus trap, your keyboard navigation isn't working properly. I know we've got a solution for this. So you can go to your knowledge base, find that information quickly, and then get that implemented as soon as possible. But yeah, look for your low hanging fruit as well in that sense. The things that are really simple to fix, like adding a bit of old text or adding an aria label, checking your roles, those sorts of things. That's really, really helpful.

- Yeah, and also from my experience also dealing with developers is that once they fix an issue once, it will become part of their development process. So they won't repeat the same mistake again in the code. So it does take a little bit of time for them to learn. So awesome. Thank you. Alright, so the next question is, okay so you mentioned that Deque helped with the review and annotation of your wire frames and designs. So the question is, is this temporary or are you eventually, are your designers gonna eventually be able to pick up that work or is this something that you'll continue to rely on Deque or other vendors to do that for you?

- Long term? No, I hope not. I think getting the right training in and using that process to gain our knowledge and build that into our process with that shift left, the goal is to just have that be something that we do every day. That inclusivity is always top of mind, that we always complete these certain things when we are building, you know, building a design. So we have all the focus indicators, you know where you are going on the pages, you've got the right names and roles and things actually identified up front. And we have a really great UX writing team at UT as well. So they've been doing a lot of help around that, like getting the wording right. Especially when you have to implement like screen reader only text because you maybe, you've not got enough real estate on the page and you can only fit, you know, a read more here button. You can add more context into that as well. So not long term, but certainly in the short term so that we can build that learning, make sure that we are confident with the solutions that we're delivering, and that that interpretation is going well through the pipeline as well.

- Right. Thank you. Alright, next question is, as a government agency, we run into funding challenges. How would you recommend integrating wire frame reviews into our process without a third party? So.

- Okay, there's so much great advice out there online. You really can get, you can do most of this by yourself. It's finding the time to dedicate to it. So if you haven't got the budget, it's gonna cost you in time because you're going to need to complete those processes yourself. But Deque is so supportive. They have some amazing checklists out there that can really help you build that into your process. And again, have an understanding of how you need to annotate your wire frames, what things need to be looking out for, like consistently, like I say, the focus of your elements as you tap through. There's some really simple things that you can start off with. So for me, I'd start small, get the basics in place and then build that up incrementally and do training as well. If you haven't got budget to do that annotation side, like do training with your teams, there's lots of great courses out there and content available. That would be my advice.

- Thank you. Alright, okay. So I think we have time for a couple more questions. Alright, so the next question is, how did your leadership support for your accessibility program happen? And what advice do you have for other organizations to help them get that level of leadership support?

- Okay, as I said, I started in November, 2020, so a lot of this was already in place. We had the vision, we had the backing, we had the right people in place. And obviously for us being the type of business that we are, this is so important to us to get right and so important so that we can gain all of that real human insight. So I don't know if I can really advise about this, but use those points about obviously how beneficial this is to your customers, how this is a human right at the end of the day, you should be doing this. This is the right approach to take. And there are so many things that you can do simply that are gonna make you more cost effective. You are gonna save money by doing this. You're gonna mitigate any risks that you have around legalities and people suing you because you've not implemented this properly. So it is really important. And I think as well, like definitely with Deque, they really helped show me that picture. I was lucky that I didn't need it. I didn't have to go to my leadership and show them all the cost benefit and all that analysis, but they've got some great information in that respect. So if you really wanna help build that business case, I'd certainly talk to Deque. And again, there's a lot of information online that you can show them about the real benefits of that, and you're welcome to still my slides and those two slides in particular, to show the benefits of that shift left and how much it's actually costing you to find these issues in production.

- Thanks, Natalie. Alright, so I think we have time for one last question and this is sort of related. So it's asking, who were the decision makers when determining when user testing was ready to do the first official audit with Deque. I know you mentioned you started, you know, right around that time when that partnership happened, but I guess, do you have any thoughts or comments about that? Go ahead.

- Okay, so we already had a great accessibility team, Accessibility Guild here. So there were lots of people already having these conversations and because it didn't really have a home at that point when we first started, the security team really helped us out because obviously they've done a lot of audits and have a lot of experience in that area. So they really helped us out to map out what we needed to do and to get that support. Our CTO Kaj has just been absolutely amazing. He helps us to drive that forward and he's always pushing me to make sure we've given those updates and to keep it top of mind. I think that's the biggest thing. So get buy in from your leadership. You need it. You need the right stakeholders and the right people in place. So you can do a cost benefit of that, if that's the main thing that drives you in that sense, but also from a customer perspective, you are gonna be in a better position and you're gonna be able to serve more people. So, it's the right thing to do.

- Definitely. Yeah, I agree. Well, unfortunately, we've come to an end of our session today. So I wanna thank you, Natalie, so much for a great session and thank you everyone for attending. Just a friendly reminder that you can always come back to the session page and watch the recording anytime. And the slides are also available on the session page. So enjoy the rest of your day and enjoy the last day of Axe-Con. Thank you.

- Thank you everyone.

- Bye bye.

- Have fun. Bye.