It gives me great pleasure to introduce this morning Roman Deuce, who's going to give us a talk on user acceptance testing. Please give him a warm welcome. Thank you very much for joining me here about my talk about automatic acceptance tests for terminal application. I thought before I get into the core of it, share a little bit how I structured the presentation. And I think in the beginning it's probably important to know a little bit of context about the project, about the terminology, what I mean with acceptance tests and how they sort of stand out between the other kinds of tests and the motivation of what made us actually write this kind of test framework. Specifically then, you know, what lessons we've learned and what we've gained. Now just quickly about me. I live in Brisbane. I've been in the open source world now for almost 20 years, been to conferences for almost 15. And thinking about it's actually kind of funny. You can do talks at conferences like for 15 years and you're still nervous. So who thinks I'd be too nervous to give talks at conferences, please do, because you'd still be nervous in 15, 20 years' time. I've worked for a while at Red Hat. And the reason why I say that some of the procedures and processes I've learned, they're basically fed back into what I'm presenting today. So that's kind of important. At the moment I'm working at an ISP in Brisbane called Superloop on a firewalling appliance. So a little bit about terminal applications. So what I mean by terminal applications, I mean the terminal can be a big thing. And what I mean by is like full terminal text user interfaces as opposed to command line tools. You might have heard about LS, LESS or GREP. These tools you invoke once with arguments. They do their thing and then basically quit. Terminal user interfaces are a little bit different because they have their own event loop. They give you graphical, I would almost say a great graphical UI, depending where you stand when we talk about terminals. But they basically they are their own thing. And when I say that, a former colleague of mine, Fraser, who is sitting in the audience, and I, we are big fans of terminal applications, especially mail user agents. We love MUT. And so we have had some great new ideas about how we would approach this field of mail user agents in the terminal. And you know how it is. We basically decided to write our own in order to implement these ideas. And that has now been going on the way for like two years, and we called it Pure Bread as a little homage on MUT. And just to bring out the point, just a quick demo. So that's on the side. That's basically the user interface. You can scroll through. You can look in what has happened on the LCA speaker mailing list. You can search mails. You can tag mails. Think a little bit like Google Mail on the terminal. And when we come to testing these terminal applications due to the fact that they have basically their own life, there is not really a sophisticated test framework around, especially not when we started it and current search results are diminishing small. So in order to like contextualize a little bit what I mean with acceptance tests or testing, let me basically zoom out when we test applications and start with the biggest approach, which is basically testing an application as a user would use it. And we call them, you know, system of black box testing. You might have heard about Selenium and Puppeteer. These are like test frameworks for web applications. And you can basically when you write tests with this framework, you can basically see how Selenium is basically using your web application as a user would. You know, like you can input text. You can click buttons and navigate through the application as if basically a user would do that. Just much faster. Basically those tests have a little bit of a setup cost because it's not really apparent on how to manipulate the application itself to get it into a state so that you can easily test it. So the setup costs are typically higher. And due to the fact that you basically have two systems like the test itself and the application itself running side by side, you basically run into like these little intermittent failures, which I want to talk about later a little bit about the pain points basically. But these are sort of the, I would say, properties of what we mean by system of black box testing. Now if we zoom in a little bit and look at components of applications, right, then if we test those, we have a little bit more control over them on how we set them up and how we manipulate them so that we can actually test them. And typically we would refer of those tests as functional integration tests. Now they are typically less API dependent and they are less maintenance effort in order to change those if the internal API changes, but they are still kind of dependent on how they are hooked up. So they are a little bit better than unit tests, which I come to now because the unit tests are typically testing the smallest unit in your application like functions and methods. And they're really quick to write and fast to execute since they are like testing the smallest possible thing. And due to that fact, they are typically they help finding and debugging problems very quickly. As in the opposite side, due to the fact they are so dependent on API changes, they can be a nightmare to maintain though. And now what do I mean with acceptance testing? When we start of implementing certain features, we typically come down to saying these features should fulfill certain requirements. And these requirements can be amalgamated as a list of criteria. So we call them acceptance criteria. And what I mean by acceptance tests is that we want to basically test the application as if a user would be using the application. And you could basically tie this into user stories. So as a system admin, I would like to perform this in this task. And that would be an acceptance criteria for this test. Now obviously this can be carried out by any form of unit integration or system test. When I say acceptance testing in this talk, I mean system tests. And when we categorize the testing itself, we also have to look at how the testing is carried out. Just quickly, from my experience, they can be carried out by a human manually as well as automated. I certainly have seen unit tests carried out by humans as well as being completely automated. They are obviously ups and downs. In my experience, humans are really good at finding subtle differences in probing applications. They are not really good at repetitive work. Whereas the automated things can be infinitely repeated without typically any problems. And I want to prove my point. So on the left we have the manual testing. On the right we have the repetitive automated testing. And I'll let him shoe up this melon once again and move on. But I guess you see what I mean with automated versus manual. Now why would we want to embark on, you know, automatically test our mailer application or terminal applications? Now as I already indicated, Selenium for the terminal unfortunately doesn't exist. And there is plenty of support if you want to test your command line apps. But there is almost no support when you want to test your CLI, sorry, your text UI. Now the typical approach is, well, how about unit testing your terminal application UI? And if you factor out parts of the application, it is kind of straightforward. But when it comes to side effects, testing that with unit tests, it's kind of not really working because typically what you end up with is like you mock out certain side effects and as much as the application matures, right, the more sort of disjunct those tests become. And then there's this phenomenon I've experienced is that you have this large test base and you think like, hmm, I've now changed this feature, but is this test actually still valid? And you sort of drag it along by make it pass, but you're kind of not really sure if it still applies but it's still in there and it's basically code you have to maintain. So it's kind of a tricky story, which basically brings us to TMAX. TMAX calls itself a terminal multiplexer. But what it sort of in layman terms what it is is that it allows you to run multiple applications at the same time in a terminal. And people of you who might have known New Screen because it's a similar program, what I want to quickly show you is a TMAX session. So I'm just emphasizing this a little bit. Now what you can see at the base here is basically what is so-called a hard line. And it's basically from TMAX a way to identify first the session on the left. That's the first session I've opened. And what you then can see is basically applications running in each window. And as you can already see, I have like two shells open, bash, and in window three, bash, as well as an application called WeChat, it's an RLC client. It's running. You can change to it. As well as our terminal mailer. And what TMAX lets you do now, you can detach from this session. It's still like all your applications are still running. You can attach again. And again, they are still running. And just for the sake of this talk, I also want to point out that, as I said, this is a session. The session is controlled by a TMAX server. And the TMAX server can control multiple sessions. And in each session, you have windows where your applications are running. This is important because I'll be talking more about, you know, TMAX sessions and TMAX server. So I hope you follow me so far. And the reason why we chose TMAX was it was the ideal tool to implement automated testing. Because A, you can remote control the terminal. You can remote control every application in a window in TMAX. You can send input events to these applications. It even supports mouse events. So it's easily scriptable, basically. And the second point is also very important. It allows you to screenshot or hard copy the contents of each window. And that is important because, well, you want to know what's going on, right? And from an automated point of view, you want to, you know, assert against the contents of what's going on. And which brings us to the point on how we use TMAX. Because when we started off, we had this implemented basically in our test code. And we factored it out as a separate library. Now we call the Tasty TMAX because there's a little bit of a back story. And the Haskell world, all of it is implemented in Haskell. In the Haskell world, Tasty is a test framework which allows you to extend it with plug-ins. Now this library does not specifically use the Tasty interfaces and Tasty library, but it leans towards using that test framework. And so we basically factored it out. It's now a separate library. It's called Tasty TMAX. It allows you to implement acceptance, automated acceptance tests for your application, for your terminal application. We use it to test our application. I want to just quickly show you a demo of how this works. And I hope everyone can see that. I'll make this a little bit better. So please bear with me. Obviously, it's all text-based. And due to the fact that I have the font so high now so that people also in the back can see it, you might not be able to see everything. But what I'm going to do is I basically run two tests. And I will show you that I've manipulated one test so that it fails at the end. So you see what I've already tried that it hopefully works. You can see part of that over there. And in the second terminal down below, I will attach to the running test and to observe what's going to happen. Obviously, in CI, you'll probably not do that. But as a developer, it's a nice thing that you can actually observe what's going on. So I start this test on one side. In another terminal, you can see down here that there are sessions created. And now I'll attach to one. You can basically now see how the application is tested through the test framework, similar to basically how you would observe Selenium testing your web application. And due to the fact that I slipped in a little problem, the end of the test fails now with a raw dump of the terminal as well as a last screen capture to see what happened and what the last state of the application was. I want to quickly go about how we've structured the test framework so that you can follow off what has actually happened. Basically what we do is we when you use Tasty T-Mux, the Tasty T-Mux library starts a session and a server, and you need the server which controls the session. We also additionally start a keep alive session, and that ties into the pitfalls I will mention later. We prepare a test environment, and that is application specific. So that is probably something you will do differently than we do. So we, for example, set up mailders and set up not much anything which is specific to our application. We just do a cleanup code so that if something happens, all resources are cleaned up. That's kind of how most test frameworks go about their daily job, nothing exotic here. I wanted to quickly introduce some example code. Now, how many people are Haskell developers? All right. So I have a little proposition for the ones who didn't raise their hands. If you're curious about Haskell, if you're curious about functional programming, join us and experience and learn some real world Haskell. It will teach you interesting lessons about how to go and work with other languages like TypeScript and Rust. So if you're curious, please join us contributing to our application, PureBread. For the ones who raised their hands, also join us and make the program a bit more awesome. So just a quick sort of rundown. I don't want to bore you too much with code examples here, but I just want to point out that this is an example from the library which is thankfully all written down if you would try it out. Thanks to Fraser. But basically what I want to point out, it is a way to refer to the environment which has been set up, and each test basically sends input, has an assertion next to it what we expect on the next screen shot, as well as has an annotation of the rundown when the test runs. And while the teardown is basically, again, nothing exotic, it basically kills all sessions, removes any resources which have been set up, and again, that's sort of standard practice for test frameworks, but there's nothing different here. One of the things I want to highlight is like the process. We at Tier 2 is that we run all acceptance tests before we merge any code so that you can basically figure out whether a merge request actually regresses the application, and so far that has been proven very useful. Now I want to quickly come to the lessons learned, aka the pitfalls. We've got our fair share of timing problems, and I think if you've been automated tests as well, then you've probably come across these problems as well because they are sort of the most common. And as I said, it's typically due to the fact that there are two systems sort of running side by side, and they need to sort of catch up with each other, and so when they get out of whack, you can run into these problems as well. In the web, it's typically if you see if the application shows, for example, a message which then disappears, that's a great source of where you can run into timing problems that the test isn't aware that first of all the message appears and disappears for a certain period of time. And unfortunately, I really don't have like a silver bullet to tackle this. In my opinion or in my experience, the best way is really to have like a deep understanding and to have developers who have the time to get a deep understanding of the application because you do need to reason about the entire components of your application in order to create a hypothesis of why this test is randomly failing. There's a technical, I mean, we can't deploy them for our spare time project Pure Bread, but there is a little bit of a technical solution too. You can sort of throw it at if you have access to a lot of VMs, and that's sort of a churn CI set up where you can basically run the tests in a different order randomly at a different speed. We ran into this one which had something to do with T-Mux changing its compatibility between versions. And this one was actually interesting because I've put a little example down here. So you can see the control characters used by T-Mux or printed by T-Mux before and after, and these were used to basically print Pure Bread with a background. And so in the old T-Mux set up, it was sort of mangled together, the foreground and the background colors, whereas the new one separated them sequentially. And what that meant for the library, it would have to basically be aware of either or and match against either or. And that happens because Travelers is still using fairly old distributions with a stable set of programs, and among them is like an older version of T-Mux than you probably have on your machine. We had races, and the reason before why I sort of outline how sort of T-Mux works between the server and the session is because we had races with the set up and the tier down between tests. And it came to be that initially we were running each test in its own session and killed the session, which also killed the server, and the next test coming along, creating a new session and creating a new server. And this overlapping between stopping and killing a server, including the session and starting it up again sort of meant that these were like stumbling over each other and causing random failures. And I should note that in order to avoid these races, we basically now keep a live session running for the entire test time and basically clean it up once all tests have run. And there's this fair share of environmental differences, and due to the fact that most of our environments, whether they are in CI or locally, they are never really pure, you will probably always run into them, and for the purpose of this talk, especially for terminal applications, be aware that certain terminals have different support for different amount of colors. So some terminals only support 16 colors, 256, 16 bit, et cetera. So be assertive of what terminal you actually support, and that means what type of colors you're actually asserting against. Then there's this fair share of environment variables, which can, you know, expose certain programs to your test and your application you're maybe not locally aware of. And last point is this is an interesting thing, the line wrapping in a terminal, just to make my point. Every shell has its so-called PS1. You can see on the left-hand side the directory you're in. Sometimes you show, for example, sometimes you show, you know, the Git branch you're in. Basically, my point is that this line can be arbitrary long. It can be arbitrary long on your CI server. It can be arbitrary long on your local environment. And that, since all of it is text-based, that can already chew up a fair amount of characters. Now, in your test case, if you assert against a continuous line and that line gets wrapped at the end of the terminal, it can mean arbitrary failures because in the CI, the line is maybe longer or locally the line is shorter. And so you have to be aware of that. And TMAX has a little switch you can invoke a capture with. And it's called minus uppercase J to join all lines. And that allows you to basically assert against continuous wrapped lines. And that's part we put into library 2. Not specifically for our purpose. We didn't run into this specifically. But I do remember from experience that hardware can be a factor 2. And that goes back to my time at Red Hat. I remember there were versions of OpenStack which was provisioning machines with a slightly off MTU on the network interfaces. And the weird thing is that in your test, you experience dropouts and timeouts and you wonder what the hell is going on. And my point is that whether it's a virtual machine, whether it's like firmware, it's all software. And they can't like rarely but they do can also have effects on your application tests. So for the purpose of completeness, I wanted to point this one out too. And since we were at the pitfalls, I thought I also mentioned of what you can do to basically prevent it. Now protocoling is not really a prevention per se, but it is certainly important to know what it's going to cost you to fix it. And so one thing I wanted to point out that if you do have random failures in your tests, that it's certainly important to protocol all these failures. So you can basically figure out whether the amount of failures are really worth fixing if you have like a tight budget, for example. Little handy tip, attach screenshots to the bug report as well as, you know, log output because the chances if you put in links is that the one you're linking to has probably changed in a year and moved on to some other URL and then the bug report is kind of useless because it's pointing to a resource you can't access anymore. We do this and again, like that was something I've learned at Red Hat that have CI run all the tests before you actually merge because it's much easier to figure out that something regresses your application when you actually merge the pull request rather than later. And one other thing is it's good to run them, you know, regularly so that you actually find random failures. We are running ours now and Travis like, you know, every day and they're like sometimes random failures popping up we have to look into and find out why they happen. That goes basically back to what I already mentioned with like a churn CI setup. If you have access to, you know, these kind of machines, certainly go, you know, use them and utilize them to, you know, churn through your tests in order to figure out whether your tests are susceptible to, you know, different order and speed execution. I also want to point out that especially for a situation where, you know, budget is tight that it is important to have the time to debug certain failures because not only will this make you a better engineer in order to reason about your systems and to reason about like the Linux system as a whole or any kind of system, it also improves a way of how quickly you can debug problems in the future and that is certainly a really important thing to have. Quickly about what we gained though and that was actually like for us as a small project or a spare time project, really, you know, massive, can basically ship when CI is green and that ties into that you're confident enough that, you know, that the pull request doesn't request your application and where this ties into as well is that if you're in the sprint session and you have to estimate some piece of work knowing that your estimate is probably not influenced by someone else's regression is really good, especially that it makes you confident that your estimate doesn't, you know, go out of bands massively as opposed to, you know, patches being merged right in the center and later on you get like the test feedback of maybe they are causing breakage. That is specifically to our acceptance tests, specifically to system tests and black box tests, sorry, that internal changes don't really affect the maintenance of the system tests because you can still reflect at large if the feature implemented basically stays the same, well, you expect that it still behaves the same even if it has like more efficient code and that is a really nice thing to have. The other thing is that system tests are automated system tests, sorry, automated acceptance tests can be really fast and I remember I always had conversations with people, yeah, but they take four hours for a mature application or they, I don't know, take six hours and I mean fast here as compared to what's the alternative because the typical alternative is we do have testers who test our application and that can take weeks, right, so you always have to see this in comparison to what the alternative is and the typical alternatives I'm aware of are not as fast and not as repetitive. And as I already indicated, there are obviously benefits to automation in itself, right, so, you know, like you can repeat them as often as you want. Automated things can be optimized to be faster, to be more efficient and, yeah, they can be infinitely repeated. We did a little look around of what's out there called the competitors. There's a little bit of enterprise stuff around there. I found a library called liptmux implement in Python, so if Python folks are out there, they might want to look at that. Some seem to be a bend, some seem to be perhaps still maintained, so you've got to judge yourself whether you want to adopt it or not. Some future ideas we do have, especially they should be done almost up to the next LCA because everyone's now joining our project here, I think. Just kidding, but so we have some ideas about at the moment we use red axis and they behave differently because they follow different implementations and that is kind of a yucky thing. An optimization idea is to run all tests in parallel would be really good. And one thing I was thinking of was that at the moment we are basically snapshotting the terminal sort of every x milliseconds and sort of pull the state and I wonder if there's not a better way to reliably figure out if the terminal has been repainted. I'm aware there's some kind of way to monitor the terminal from tmux, but I'm not sure if that is really a good way to figure out if the terminal has been repainted. Quick some references. The top link is the link you want to go contributing and learning about Haskell, obviously. Second link is our library. Please join plenty. There is a thing when we were talking about acceptance tests, there is a thing about behavior-driven development. You might have heard about Cucumber. It's a bit of a I would say it's become a little bit of an enterprise thing that you can basically write a story down and it's something you can give your manager who understands the story. It's sort of a similar feel that you can basically communicate what you're testing with your stakeholders. Email. Fraser is more into Twitter than I do, but yeah. And that is basically the end of my talk and I'm accepting questions. Thank you. I just wanted to quickly say thank you to Fraser who helped me a lot putting that talk together and also helped me a lot learning about Haskell and mentoring me through the entire time. Basically, I want to play it forward. Whoever wants to join, as you can hear, we're desperate for contributors, but whoever wants to join and learn about the language, please come and join. One question because you were talking about regexes and those problems. Is there anything like text, finite state machine, and Haskell? I'm not specifically aware of Fraser. Are you aware of... It's certainly something to look into, especially if we think about, you know, have a better representation of what the terminal looks like and how we assert certain expectations against the terminal. So yeah, there's definitely room for improvement. Thank you all for coming. No, there's one. I was going to say, could you test with something like Parsec or C instead of regex? We had this idea too that maybe it would make sense to pass the output into sort of a representation and have a possibility to assert against that, which would probably provide a much richer way of expressing yourself against what you're expecting. Something like a DOM or something. I was thinking you'd go... Can we see CSS selected? It depends on who was joining us. Just like to encourage everyone to give another thank you to Roman. And as an appreciation, there's one there as well. Thank you. Thank you