Automated tests seem like a natural fit for a world that is pushing for shorter cycle times and more reliable delivery of software and hardware. Tests increase our comprehension while we write them. They increase our confidence when we run them. They can help us identify problems earlier and more effectively. Yet, automated tests come with real costs and with hidden risks. This talk will use experiences in the test automation of the Jenkins project to illustrate some of the costs of test automation and share insights how we can recognize those costs, manage those costs, and increase the value of automated tests. The talk will use examples from the Jenkins project of risks that come with our increasing reliance on test automation. Issues that are missed, problems that are not detected, and use cases that are not considered will all be viewed through the lens of test automation.