Open source testing tools requirements
The most popular tools for test automation are Selenium, Appium, Cucumber, Jmeter, Postman..
Testing with Open Source tools requirements are:
1. Hardware Requirements
A powerful Macbook needed for automation engineers. Because You will need lots of things for development and debugging test automation. Mac-mini, USB extender, iOS and Android Devices. Some guys still have to code with VBscript and use Windows laptops. It’s not your fault :(
Conclusion: There must be a different and common way of doing this, we can’t dedicate all devices to all engineers. First of all it should be in a shareable environment.
2. Software Installations & Configurations
Installation of Appium and all dependencies on your computer for mobile test automation development
Conclusion: I am not a mobile developer, do I really have to install so many things on macbook? I am installing more than an Android or iOS developer. How will I keep them updated? How do I upgrade to the new SDK version?
3. Get Required Access Permissions
Test Database Access Permissions
Backend System Access Permissions
Git Repository Access
Test Management Tool Access
Issue management Access
Notification app Access
Conclusion: If I have just started this job, I may not even know which systems I should have access to. It will also take a while for anyone new to the team to open, close, or update account / access to these systems. there must be a definitive solution to doing this in a different way.
4. Inspect Existing Test Automation Project
Automation Project’s Programming Language and Framework.
Supported Platforms (iOS, Android, Web, API, Hybrid, Desktop..)
Test Automation Scenarios
Merge and branching strategies
Conclusion: Suppose that You feel better with Javascript automation projects. But The existing project had been written in JAVA. It will take time to learn. You can’t change the framework or language. There are thousands of lines of codes and a sprint has been started. Time is not enough for all and the team is not ready for big changes. There are common challenges in your code.. False Positives & False Negatives (Flaky Tests) And your existing test automation framework doesn’t support API testing. Let’s over engine this issue. Where is the right place for your mobile automation code? Should it be in the app or outside of the app?
5. Application package to be tested
You have to test the right packages at all times. You have to access APK(Android) and IPA(iOS Real Device) files on mobile projects.
Package from the Latest Branch / Specified branch
Conclusion: Will I compile and create the application package to be tested, or will it come to me automatically or manually from the continuous integration tool?
6. Emulator vs Simulator vs Real Device
Emulators and Simulators are free and they have no duration limitations like cloud services. But can’t replicate real user conditions like hardware and software configurations, work slower than real devices, consume a lot of memory and CPU on your computer.
Emulators / Simulators are good at Unit Testing, Static Code analysis, development. They are weak at UI Regression Testing and Compatibility testing
Conclusion: Real devices provide test runs under real-world conditions. If your test is passing on an emulator / simulator that doesn’t mean it will pass on Real Device.
7. On-Premise (In-House) vs Cloud Device Farm
You need a device farm for compatibility testing and high device coverage.
Conclusion: Cloud farm has got limitation, pricing and slowness problems. Can I believe that the mobile phone on which I ran the test is real and only dedicated to me? I prefer other users not to access my application and automation code.
8. Interactive Live Mobile App Testing on Real Device
We are going to automate 100 percent of our regressions.
Automation and manual test engineers are independent teams.
Conclusion: Manually exploratory testing is very important now and will remain so in the future. There is a problem of device sharing between manual and automation test engineers. Devices in on-premise device farms also have the same problem. You should not separate Manual and automation engineers. The goals of both teams are the same. Measuring the quality of the software. There is actually 1 team. You should not automate everything, Welcome to the real world.
9. Start to Automate
Inspect ID, XPATH,.. of the mobile elements on APP and write your automation code on IDE. ()
Element locator problems (There is no ID or Name, Dynamic ID’s,..)
Conclusion: Using Appium with your favorite development IDE (IntelliJ, Pycharm, WebStorm, VisualStudio, Eclipse, Ride.. ) Can't we do this with a single application instead of 2 different client applications with hours of installation and configurations? Should we centrally manage Appium server and client applications? Can I get support from the development team for element locator problems? HTML concepts and basic programming knowledge is required. Reusable elements can help you.
10. Integration Requirements
JIRA issue management / Slack notification / Email / Jenkins..
Auto-fill and submit test run Defects to Jira
Basic reporting with Email
In Real Time Get Slack notification messages for test execution.
CI tool plugin to trigger Test Run.
Git integration
Cloud Device Farm integration
Conclusion: Developing dozens of integrations will require serious effort. If I cannot solve this work with open source resources in a short time, I may have to replace the complete test automation framework. We may even need to get test consultancy.
11. Code Quality
Use PageObject Pattern
Fix Static Code Analysis issues.
Apply SOLID..
Write BDD TestCases
Apply GitFlow branching model
Conclusion: What is expected of me? I am not a developer..
12. TestCases in BDD (Cucumber)
Given-When-Then formatted Acceptance criterias needed
Conclusion: JIRA issues are not in BDD format. How can I adapt the current format to this?
13. Test Data problem
Reading Data from Database/File/..
Writing Data to Database/File..
Create Test Data sets for your test automation suite
Use Data Fakers
Can’t generate Test Data, Mock the middleware
Conclusion: The test data problem will always be your problem. The important thing is how much support is built into which tool. Data-driven testing can help you.
14. Debugging Features
Check your test’s screenshot, video on each step.
Check the console logs, application logs.
Debug your test run locally.
Conclusion: Debugging the test run is very important. Without debugging, it’s like a blind man driving a car.
15. Run Your Test
Check the availability of the devices / browsers
Multi device / browser & version selection
Check the availability of CI / CD tools
On demand or Scheduled test running
Separated Test Scenarios
Running Test Code on Docker Container
Conclusion: Where is my project built? Where is the test running on? How can I run it locally or remotely? How can I watch the stage of the test?
16. Running on Different devices (PARALLEL) / OS / Versions
Check that all test automation results on all popular devices / versions available are as you expect.
Running Tests with Selenium Grid
Using Local or Remote Devices.
Emulator / Simulator / Real Device again.
Conclusion: If all tests take two hours to run on 1 device, it should take an average of two hours to run on 100 devices. Other options are unacceptable and unexpected for me.
17. Reporting test results
Test Case / Test Run report
JIRA Defect Reporting
Conclusion: Reporting is not a nice to have feature, it is a must.
18. Community Support
If you started with java, there is a lot of community support for issues / libraries. You may not have much chance for other programming languages.
The incompatible tool versions should conflict with each other and You should get fail when you want to run your test
You should get no support / response on some issues with open source. You are alone.
Risk of violation and license restriction risk.
and more…
Learn more about Review & Compare Testing Tools
