Testing CLI Utilities

Rate this content
Bookmark

Ever wondered what is the best way to end-to-end test your custom command line utilities? In this talk Florian Rappl will give you some insights what you can do to automatically verify your CLI tools and avoid regression.

- Introduction: Why test CLI tools

- Challenges: File system pollution, network and database issues, environment variables

- Demo: Showcase issues with a demo CLI tool - Solutions: Test plan implementation, choosing the right level of containerization

- Demo: Show solution using the previous CLI tool

- Conclusion

This talk has been presented at TestJS Summit 2022, check out the latest edition of this JavaScript Conference.

FAQ

Florian is a solution architect at SMAPIOK, a company based in Munich, Germany, specializing in IoT and embedded computing. They focus on building digital transformation projects, particularly using distributed web applications.

Florian is an open-source enthusiast and has been awarded a Microsoft MVP in the area of development tools for the last decade. He has extensive experience with .NET, JavaScript, and TypeScript in web development, and has written articles and books on micro frontends and frontend application development.

Testing CLI utilities is challenging due to the need to handle inputs and outputs, manage asynchronous processes, and coordinate system resources. However, it is appealing because CLI utilities often play a crucial role as an intersection point in applications, making them important to test for reliability.

Florian suggests using sandboxing for file system operations, especially when running multiple tests in parallel, to avoid race conditions. He also emphasizes the need for performance optimization in testing, such as using temporary directories to reduce setup times for multiple tests.

Florian employs Jest as a test runner, Playwright for browser automation, and TypeScript for writing tests, running these in Azure pipelines for continuous verification. He also utilizes strategies like creating template directories to speed up testing and ensure consistency.

Proper port allocation is crucial in testing to avoid conflicts when multiple tests demand the same service on the same port. Florian uses a port allocator to dynamically assign available ports, ensuring that tests run smoothly without port conflicts.

Florian Rappl
Florian Rappl
34 min
03 Nov, 2022

Comments

Sign in or register to post your comment.

Video Summary and Transcription

CLI utilities are important to test because they act as an intersection point between different parts of an application. The main challenge in testing CLI utilities is performance, which can be improved by using temporary directories. Managing ports and resources is crucial to avoid conflicts when running multiple test suites. The test context ensures that processes run in the correct context, including the use of the right directories. Running tests on different configurations helps identify compatibility issues and provides comprehensive test coverage.

1. Introduction to Testing CLI Utilities

Short description:

Welcome to the session, Testing CLI Utilities. CLI utilities are important to test because they act as an intersection point between different parts of an application. They can access network resources and interact with the system locally, which requires proper coordination and sandboxing. These challenges make CLI utilities particularly interesting for testing. For example, PyCLI is a utility that helps with web development tasks like scaffolding and running debug processes.

Welcome, everyone to the session, Testing CLI Utilities. I hope you are as motivated as I am to get started.

Before we begin, let's have a look at my person. Hi, I'm Florian. I'm a solution architect at a smaller company based in Munich, Germany, called SMAPIOK. We are mostly doing IoT and embedded computing and we are specialized in building digital transformation projects, especially using distributed web applications. Also, I'm an open-source enthusiast. What does that mean? Well, I've been awarded a Microsoft MVP in the area of development tools for the last decade. I spent most of my time doing projects in .NET, JavaScript, TypeScript, web development space. I'm writing a lot of articles and I've also written one book already about micro frontends and I'm currently writing another book about development of frontend applications with NoJazz, so be sure to get a copy. But enough about me, let's just jump right into a topic before we run out of time.

So CLI utilities, what makes them difficult to test? What makes them also appealing to test? Well, first of all, I mean, CLI implies they're running the command line, so this is on the one hand good because spawning a terminal process is always quite easy and in some sense also easy to work with, much easier to work with, for instance, a graphical user interface. On the other hand of course, you need to deal with some things like for instance, receiving the standard output or also placing some inputs on the standard in stream. And you of course need to coordinate that and have your asynchronous processes right. CLI utilities often provide, let's say kind of an intersection point between two parts of an application so they are quite important to test to get right. And we are relying a lot of CLI utilities, so I mean having them working reliably is of course always what we strive after. Anyway, these CLI utilities like any other application, they might also access some resources. For instance, of course, network resources. And yeah, I mean, you may want to mock these, you may need to run local services and you may also need to coordinate these resources. So that's something you need to keep in the back of your head. Also of course with each step that you do in a mocking direction, of course you remove a potential source of error for a later run and you will need to accommodate for that. The most important area though is of course what happens locally on the system. So for instance on the file system, read and write operations need to be sandboxed properly. If you run especially multiple tests in parallel, you can't just go in blindly and say, whatever does these utilities that I'm testing are doing, I just trust them that they always of course work in, I don't know, dedicated directories and that there are no race conditions or whatever, really running them in parallel. And that of course applies to any kind of system resource that they access. So we've got already a set of challenges but also a set of things that make CLI utilities particularly interesting for testing.

Right, so going a little bit further and thinking about some challenges that arise in that, we've already heard about input output serving content was already briefly touched just to remind you. So for instance, let's say what we will be testing is so-called PyCLI. It's a little utility that helps with a couple of web development related tasks. One of these is scaffolding, but another one that you would do quite after the scaffolding is running, for instance, a debug process with it.

2. Verifying Port and Content, Handling Fragmentation

Short description:

The CLI utility opens a port and serves output on that port. To ensure correctness, we can ping the port and use utilities like BlayWrite to access and verify the content. Fragmentation is a common challenge where the utility places content in multiple directories. This can be avoided by cleaning up after each test and ensuring test isolation.

And this debug process is actually opening a web server on your local machine. Now we want, of course, to verify that this has been opened successfully. And so of course, this part of the CLI that it opens some port is something that we need to consider. Now, how do we ensure that the correct port was opened, but also then, of course, that the output served on the port is right? You need to, of course, have all those questions answered.

In our case, what we did, of course, is we ensure we have just some ping on the port, that the port is alive, but then afterwards what we can do is using utilities such as BlayWrite for actually accessing the content, reading out it, and verifying it against, let's say, an expectation source that the actual reserved resources is the same.

Now, fragmentation is something that will appear quite often, which means that a CLI utility might, let's say, place content that it, for instance, creates in a couple of directories. And you don't want that to happen so often, because first of all, you need to clean up after each test. And second, you need to ensure that the test, of course, stays isolated, right? That you can run, for instance, multiple test cases in parallel or multiple test suites. And if you don't have control over where the utility that you want to test is placing files, you might run into these race conditions that I mentioned earlier.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Atomic Deployment for JS Hipsters
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
Atomic Deployment for JS Hipsters
This Talk discusses atomic deployment for JavaScript and TypeScript, focusing on automated deployment processes, Git hooks, and using hard links to copy changes. The speaker demonstrates setting up a bare repository, configuring deployment variables, and using the post-receive hook to push changes to production. They also cover environment setup, branch configuration, and the build process. The Talk concludes with tips on real use cases, webhooks, and wrapping the deployment process.
Effective Performance Testing to your Server with Autocannon
TestJS Summit 2021TestJS Summit 2021
36 min
Effective Performance Testing to your Server with Autocannon
Top Content
Tamar is an experienced code writer and architect with expertise in Node.js. Performance testing can be confusing, but understanding terms like throughput and the 99th percentile is crucial. The 99th percentile is important for making commitments and ensuring customer satisfaction. AutoCanon is a powerful tool for simulating requests and analyzing server performance. It can be installed globally or used as a library in Node.js. Autocannon is preferred over Gatling for performance testing and can be integrated with end-to-end tests in Cypress.
Delightful Integration Tests With Testcontainers
TestJS Summit 2022TestJS Summit 2022
21 min
Delightful Integration Tests With Testcontainers
Top Content
Testing is crucial for development and production, with integration tests becoming more popular. Test containers is a library that integrates with Docker to create reliable test environments. It is flexible and can be used with various frameworks and test libraries. The IDE setup involves configuring the container and connecting it to the application. Test containers can be used for complex operations and allows running tests with real dependencies.
Visual Regression with Puppeteer, Playwright and Cypress
TestJS Summit 2021TestJS Summit 2021
9 min
Visual Regression with Puppeteer, Playwright and Cypress
Top Content
Hello, I'm Rainer Haneckamp, a trainer and consultant at Angular Architects. In this talk, we'll explore visual regression testing using tools like Puppeteer, Playwright, and Cypress. We'll learn how to use Storybook and Puppeteer with Jest for visual regression testing. We'll also see how Jest and Playwright can be used together for visual regression testing. Finally, we'll discover how to use Cypress for visual regression testing. Thank you for watching!
Playwright Can Do This?
TestJS Summit 2022TestJS Summit 2022
23 min
Playwright Can Do This?
Playwright is a powerful tool for end-to-end testing, offering support for all major browsers and platforms. It provides features like parallelization, built-in waiting, and assertions. Playwright allows for running tests on multiple browsers with a single command and has functionality for generating tests and performing visual regression testing. It also enables the manipulation of the network layer and loading internals of web pages. Best practices include using short and idempotent scripts, splitting user account flows into separate tests, and cleaning up after each test case.
The Lazy Developer Guide: How to Automate Code Updates?
DevOps.js Conf 2022DevOps.js Conf 2022
22 min
The Lazy Developer Guide: How to Automate Code Updates?
Code automations can save time and effort in development tasks. There are tools and examples available for automating tasks like updating dependencies and code formatting. Automation allows teams to focus on valuable work and improves overall performance. Deciding when to automate depends on the impact and type of code. The last automated task discussed was applying translation updates to multiple projects.

Workshops on related topic

Automated accessibility testing with jest-axe and Lighthouse CI
TestJS Summit 2021TestJS Summit 2021
85 min
Automated accessibility testing with jest-axe and Lighthouse CI
Workshop
Bonnie Schulkin
Bonnie Schulkin
Do your automated tests include a11y checks? This workshop will cover how to get started with jest-axe to detect code-based accessibility violations, and Lighthouse CI to validate the accessibility of fully rendered pages. No amount of automated tests can replace manual accessibility testing, but these checks will make sure that your manual testers aren't doing more work than they need to.
Automated Testing Using WebdriverIO
TestJS Summit 2022TestJS Summit 2022
163 min
Automated Testing Using WebdriverIO
Workshop
Kevin Lamping
Kevin Lamping
In this workshop, I cover not only what WebdriverIO can do, but also how you'll be using it day-to-day. I've built the exercises around real-world scenarios that demonstrate how you would actually set things up. It's not just "what to do," but specifically "how to get there." We'll cover the fundamentals of Automated UI testing so you can write maintainable, useful tests for your website and/or web app.
JS Security Testing Automation for Developers on Every Build
TestJS Summit 2021TestJS Summit 2021
111 min
JS Security Testing Automation for Developers on Every Build
WorkshopFree
Oliver Moradov
Bar Hofesh
2 authors
As a developer, you need to deliver fast, and you simply don't have the time to constantly think about security. Still, if something goes wrong it's your job to fix it, but security testing blocks your automation, creates bottlenecks and just delays releases...but it doesn't have to...

NeuraLegion's developer-first Dynamic Application Security Testing (DAST) scanner enables developers to detect, prioritise and remediate security issues EARLY, on every commit, with NO false positives/alerts, without slowing you down.

Join this workshop to learn different ways developers can access Nexploit & start scanning without leaving the terminal!

We will be going through the set up end-to-end, whilst setting up a pipeline, running security tests and looking at the results.

Table of contents:
- What developer-first DAST (Dynamic Application Security Testing) actually is and how it works
- See where and how a modern, accurate dev-first DAST fits in the CI/CD
- Integrate NeuraLegion's Nexploit scanner with GitHub Actions
- Understand how modern applications, APIs and authentication mechanisms can be tested
- Fork a repo, set up a pipeline, run security tests and look at the results
Security Testing Automation for Developers on Every Build
GraphQL Galaxy 2021GraphQL Galaxy 2021
82 min
Security Testing Automation for Developers on Every Build
WorkshopFree
Oliver Moradov
Bar Hofesh
2 authors
As a developer, you need to deliver fast, and you simply don't have the time to constantly think about security. Still, if something goes wrong it's your job to fix it, but security testing blocks your automation, creates bottlenecks and just delays releases, especially with graphQL...but it doesn't have to...

NeuraLegion's developer-first Dynamic Application Security Testing (DAST) scanner enables developers to detect, prioritise and remediate security issues EARLY, on every commit, with NO false positives / alerts, without slowing you down.

Join this workshop to learn different ways developers can access NeuraLegion's DAST scanner & start scanning without leaving the terminal!

We will be going through the set up end-to-end, whilst setting up a pipeline for a vulnerable GraphQL target, running security tests and looking at the results.

Table of contents:
- What developer-first DAST (Dynamic Application Security Testing) actually is and how it works
- See where and how a modern, accurate dev-first DAST fits in the CI/CD
- Integrate NeuraLegion's scanner with GitHub Actions
- Understand how modern applications, GraphQL and other APIs and authentication mechanisms can be tested
- Fork a repo, set up a pipeline, run security tests and look at the results