|
| 1 | +--- |
| 2 | +layout: classic-docs |
| 3 | +title: "Testing CircleCI Orbs" |
| 4 | +short-title: "Testing Orbs" |
| 5 | +description: "Starting point for Testing CircleCI Orbs" |
| 6 | +categories: [getting-started] |
| 7 | +order: 1 |
| 8 | +--- |
| 9 | + |
| 10 | +This page describes various testing methodologies you can test use to test orbs. |
| 11 | + |
| 12 | +* TOC |
| 13 | +{:toc} |
| 14 | + |
| 15 | +## Introduction |
| 16 | + |
| 17 | +CircleCI orbs are packages of configuration that you can use in your configurations to simplify your workflows and quickly and easily deploy application and orbs in your workflow. When creating orbs for your workflow, you will want to perform testing to ensure your orb meets your specific needs before deploying and publishing the orb. |
| 18 | + |
| 19 | +You can think of orb testing at four levels, in increasing levels of complexity and scope. |
| 20 | + |
| 21 | +* Schema Validation - this can be done with a single CLI command and checks if the orb is well-formed YAML and conforms to the orb schema. |
| 22 | +* Expansion Testing - this can be done by scripting the CircleCI CLI and tests whether the elements of the orb generate the configuration you intended when processing configuration containing those elements. |
| 23 | +* Runtime Testing - this requires setting up separate tests and running them inside a CircleCI build. |
| 24 | +* Integration Testing - this is likely only needed for fairly advanced orbs or orbs designed specifically as public, stable interfaces to 3rd-party services. Doing orb integration tests requires a custom build and your own external testing environments for whatever systems you are integration with. |
| 25 | + |
| 26 | +The sections below provide details on each of these techniques for each level of orb testing. |
| 27 | + |
| 28 | +### Schema Validation |
| 29 | + |
| 30 | +To test whether an orb is valid YAML and is well-formed according to the schema, use `circleci orb validate` with the CircleCI CLI. |
| 31 | + |
| 32 | +#### Example |
| 33 | + |
| 34 | +Given an orb with source at `./src/orb.yml` you can run `circleci orb validate ./src/orb.yml` to receive feedback on whether the orb is valid and will pass through config processing. If there is an error, you
1241
will receive the first schema validation error encountered. Alternatively, you can pass STDIN rather than a file path. |
| 35 | + |
| 36 | +For example, equivalent to the previous example you can run `cat ./src/orb.yml | circleci orb validate.` |
| 37 | + |
| 38 | +**Note** Schema errors are often best read "inside out", where your coding error may be best described in one of the inner-most errors in a stack of errors. |
| 39 | + |
| 40 | +Validation testing may also be performed inside a build using the CircleCI CLI. You can directly use the `circleci/circleci-cli` orb to insert the CLI into your builds, or use the `circleci/orb-tools` orb that contains a few handy commands jobs, including performing schema validation by default in its jobs. An example of an orb that uses `orb-tools` to do basic publishing is the "hello-build" orb. |
| 41 | + |
| 42 | +### Expansion Testing |
| 43 | + |
| 44 | +The next level of orb testing validates that an orb is expanding to generate the desired final `config.yml` consumed by the CircleCI system. |
| 45 | + |
| 46 | +This testing is best completed by either publishing the orb as a dev version and then using it in your config and processing that config, or by using config packing to make it an inline orb and then processing that. Then you can use `circleci config` process and compare it to the expanded state you expect. |
| 47 | + |
| 48 | +Another form of expansion testing can be done on the orb itself when your orb references other orbs. You can use the `circleci orb` process to resolve the orbs-dependent orbs and see how it would be expanded when you publish it to the registry. |
| 49 | + |
| 50 | +Keep in mind that when testing expansion it's best to test the underlying structure of the data rather than the literal string that is expanded. A useful tool to make assertions about things in yaml is `yq`. This allows you to inspect specific structural elements rather than having fragile tests that depend on string comparisons or every aspect of an expanded job. |
| 51 | + |
| 52 | +The following steps illustrate doing basic expansion testing from the CLI: |
| 53 | + |
| 54 | +1) Assume a simple orb in `src/orb.yml` |
| 55 | + |
| 56 | +{% raw %} |
| 57 | +```yaml |
| 58 | +version: 2.1 |
| 59 | + |
| 60 | +executors: |
| 61 | + default: |
| 62 | + parameters: |
| 63 | + tag: |
| 64 | + type: string |
| 65 | + default: "curl-browsers" |
| 66 | + docker: |
| 67 | + - image: circleci/buildpack-deps:parameters.tag |
| 68 | + |
| 69 | +jobs: |
| 70 | + hello-build: |
| 71 | + executor: default |
| 72 | + steps: |
| 73 | + - run: echo "Hello, build!" |
| 74 | +``` |
| 75 | +{% endraw %} |
| 76 | +
|
| 77 | +2) Validate the orb with `circleci orb validate src/orb.yml`. |
| 78 | + |
| 79 | +3) Publish a dev version with `circleci orb publish src/orb.yml namespace/orb@dev:0.0.1`. |
| 80 | + |
| 81 | +4) Include that orb in `.circleci/config.yml`: |
| 82 | + |
| 83 | +{% raw %} |
| 84 | +```yaml |
| 85 | +version: 2.1 |
| 86 | +
|
| 87 | +orbs: |
| 88 | + hello: namespace/orb@dev:0.0.1 |
| 89 | +
|
| 90 | +workflows: |
| 91 | + hello-workflow: |
| 92 | + jobs: |
| 93 | + - hello/hello-build |
| 94 | +``` |
| 95 | +{% endraw %} |
| 96 | + |
| 97 | +After running `circleci config process .circleci/config.yml` the expected result would be: |
| 98 | + |
| 99 | +{% raw %} |
| 100 | +```yaml |
| 101 | +version: 2.1 |
| 102 | +
|
| 103 | +jobs: |
| 104 | + hello/hello-build: |
| 105 | + docker: |
| 106 | + - image: circleci/buildpack-deps:curl-browsers |
| 107 | + steps: |
| 108 | + - run: |
| 109 | + command: echo "Hello, build!" |
| 110 | +workflows: |
| 111 | + hello-workflow: |
| 112 | + jobs: |
| 113 | + - hello/hello-build |
| 114 | + version: 2 |
| 115 | +``` |
| 116 | +{% endraw %} |
| 117 | + |
| 118 | +The `config.yml` file should look like the following: |
| 119 | + |
| 120 | +{% raw %} |
| 121 | +```yaml |
| 122 | +version: 2.1 |
| 123 | +
|
| 124 | +orbs: |
| 125 | + hello: namespace/orb@dev:0.0.1 |
| 126 | +
|
| 127 | + workflows: |
| 128 | + hello-workflow: |
| 129 | + jobs: |
| 130 | + - hello/hello-build |
| 131 | +``` |
| 132 | +{% endraw %} |
| 133 | + |
| 134 | +The result shown above may now be used in a custom script that tests for its structure relative to what is expected. This form of testing is useful for ensuring you do not break any contracts your orb interface has established with users of the orb, while also enabling you to test different parameter inputs to determine how these parameter imputs impact what is generated during config processing. |
| 135 | + |
| 136 | +### Runtime Testing |
| 137 | + |
| 138 | +Runtime testing involves running active builds with orbs. Because the jobs in a build can only depend on orbs that are either part of the config or were published when the build started, this technique requires some special planning. |
| 139 | + |
| 140 | +One approach is to run jobs within your build's jobs by using the circleci CLI to run local builds using `circleci local execute` on a machine executor within your builds. This allows you to print the build output to `stdout` so you can make assertions about it. This approach can be limiting, however, because local builds do not support workflows and have other caveats. This is also powerful if you need to test the actual running output of a build using your orb elements. |
| 141 | + |
| 142 | +For an example of using this technique see the [Artifactory Orb](https://github.com/CircleCI-Public/artifactory-orb) page in the CircleCI public GitHub page. |
| 143 | + |
| 144 | +The other main approach to runtime testing is to make the orb entities available to your job's configuration directly. |
| 145 | + |
| 146 | +One option is to make checks using post-steps for jobs that you run and test, or subsequent steps for commands that you run and test. These can check things like filesystem state, but cannot make assertions about the output of the jobs and commands |
| 147 | + |
| 148 | +**Note** CircleCI is working to improve this limitation and welcomes feedback on your ideal mechanism for testing orbs. |
| 149 | + |
| 150 | +Another approach is to use your orb source as your `config.yml`. If you define your testing and publishing jobs in your orb, they will have access to everything defined in the orb. The downside is that you may need to add extraneous jobs and commands to your orb, and your orb will depend on any orbs that you need for testing and publishing. |
| 151 | + |
| 152 | +Yet another approach is when you run a build, publish a dev version of the orb, then do an automated push to a branch which uses a config that uses that dev version. This new build can run the tests. The downside with this approach is that the build that does the real testing is not directly tied to its commit, and you end up with multiple jobs every time you want to change your orb. |
| 153 | + |
| 154 | +### Integration Testing |
| 155 | + |
| 156 | +Sometimes you will want to test how your orbs interact with external services. There are several possible approaches depending on circumstances: |
| 157 | + |
| 158 | +* Make your orb support a dry-run functionality of whatever it is interacting with, and use that mode in your tests. |
| 159 | +* Do real interactions, using a properly set up test account and a separate repository that runs those tests using a published dev version of your orb. |
| 160 | +* Spin up a local service in another container of your job. |
| 161 | + |
| 162 | +## Orb Testing Best Practices |
| 163 | + |
| 164 | +The most significant issue when testing orbs is that it is not straightforward to push a new commit to a repository containing orb source code. This is because orbs are interpolated into an expanded `config.yml` at build inception and may not have the newest changes to the orb contained in that commit. |
| 165 | + |
| 166 | +There are several different approaches you can use to test your orbs (for example, using inline orbs or external repositories) to ensure orb compatibility with the CircleCI platform. CircleCI is also in the process of developing newer ways for you to test your orbs. For more detailed information about these testing examples, refer to the [Emerging Testing Best Practices for Orbs](https://discuss.circleci.com/t/emerging-testing-best-practices-for-orbs/27474) page in the CircleCI Discussion Forum. |
| 167 | + |
| 168 | +## Orb Testing Methodologies |
| 169 | + |
| 170 | +### Testing Orbs Locally |
| 171 | + |
| 172 | +One of the easiest ways to test your orbs locally is to create an inline orb. Creating an inline orb enables you to test your orb while in active development. Inline orbs can be useful when developing your orb, or as a convenience for name-spacing jobs and commands in lengthy configurations. |
| 173 | + |
| 174 | +For more detailed information on how to create an inline orb, refer to the [Creating Orbs](({{site.baseurl}}/creating-orbs/#creating-inline-orbs) page in the CircleCI technical documentation. |
| 175 | + |
| 176 | +### Testing Orbs Automatically |
| 177 | + |
| 178 | +Testing orbs can be done at a few different levels. Choosing how much testing you want to do will depend on the complexity and scope of the audience for your orb. |
| 179 | + |
| 180 | +In all cases, CircleCI recommends that you make use of the CircleCI CLI to validate your orb locally and/or automate testing in a build. For installation instructions for the CLI see the CLI documentation |
| 181 | + |
| 182 | +For advanced testing, you may also want to use a shell unit testing framework such as BATS. |
| 183 | + |
| 184 | +## See Also |
| 185 | + |
| 186 | +* Refer to [Creating Orbs]({{site.baseurl}}//2.0/creating-orbs/#creating-inline-orbs) for more information on how to test your inline orbs. |
| 187 | +* Refer to [CircleCI CLI]({{ site.baseurl }}/2.0/local-cli/) for more detailed information on the CircleCI CLI and how to use the CLI for orb testing. |
0 commit comments