Objectives

This report is aimed at bringing into view the various methods of testing a software and fixing its bugs. Also in understanding the various methods in testing and testing tools in action when testing a chosen program during the study.

There are several requirements that are required to test a software or fix its bugs. These are the testing methods and the testing tools. In the testing methods, there are several methods that are applicable: the black box, white box and the gray box methods. But before we go into finer details about the testing methods, we must first define what is testing a software. Generally, it is a probe to check the quality of the software and to check the errors that might have arisen in the making of the software. Also checking if the software has met its requirement is a testing process.

Different methodologies have different approaches in the checking the credibility of a software. The black box method, checks if the software has met the requirements that were put down. It ascertains that the output is the expected one by viewing the user interface and examining the output. This method does not require to investigate the inner working of the program. Its output confirms it all. Apart from black box we have another method, the white box method. White box method investigates the internal working of the software. This method examines the code and identifies where the problem is and also checks if the output meets the requirements put down. The gray box method, a “hybrid method”, combines both the white box and the black box beneficial characteristics. The tester has limited knowledge of the working of the software. It is therefore effective compared to black box method where the tester just examines the output.

I used the sellAll web application to be tested with the TestNG, Yasca and Selenium testing tools. The method used for testing is the gray method. SellAll application imitates the worldwide leading online shopping application, Amazon. I settled on the SellAll web application since it has elaborated functional requirements and output can be easily examined. These functional requirements are: inputting data with ease since the user interface is friendly, the functions of the applications are easily identifiable (shopping online), the output can be easily assessed (display of items to be sold) and the task to be executed is done promptly. This application has made shopping easy and also window shopping improved since it can be done remotely. SellAll uses internet for its operation hence it can be accessed worldwide. Through this, its testing must be done thoroughly to cover all vulnerabilities.

Testing Methods and Testing Tools

Software testing is a constant present state in the ever making of a software. The process is cumbersome and rather expensive but it is inevitable hence simpler methods have tried to be put into action to try and optimize the process. For software testing to be successful, skilled personnel, excellent testing method and an appropriate testing tool must be identified. Failure to uphold these requirements can lead to more hiccups in the delivery process. More softwares are being made in the recent time hence making this testing method easier and effective is a priority. Also, more codes in a software is a chance of a bug presenting itself. Scholars have probed this matter to a great extent and provided some solutions to the existing problems in this area.

Selenium is an open source testing tool referred by several scholars. Mostly it is used to first test web applications designs before they can be used for any purpose (Holmes and Kellogg, 2006). Selenium RC eases the process of testing by closing and opening databases and then comparing the test data with data in the databases (De Castro et al, 2013). This tool is evidently standing out to be the best web application testing tool.

In their work Banabic and George called “Fast Black-Box Testing of System Recovery Code” they identified that to test the robustness or strength of a software, code injection was the way to achieve that. Though the method required a lot of manpower and a lot of injections in the code, the method tested the required areas with agility and promptness. The method is rarely used since due to a lot of manpower needed and subsequent injections, its revealing to be a little bit expensive. Therefore, for this method to be used, a specific place must be identified and the code injected must have high reliable requirements.

It is evident with most scholars that although software testing has made remarkable strides in accomplishing its purpose, it has not yet visited all corners in the area. More and more needs emerge when others are covered. For instance, the maker of TestNG, Cedric Beust, came up with the making of it due to the frustrations he got from JUNIT and decided to improve it. In his blog (beust.com), Cedric explains the TestSetup that he has improved from JUNIT and elaborates the limitations of the static methods you are forced to introduce in JUNIT. He plainly refers to these methods as evil. It is with no doubt that JUNIT has been overcame with TestNG due to lack of some functionalities. My assumption is that it is not going to be long and then something is going to be missing in the TestNG which will prompt other developers to improve on it or make their own testing tool which will have the missing function. It is like this testing need is insatiable and as languages continue to evolve, the more testing becomes broad and complicated.

Testing the SellAll Web Application

Security is the far most considered aspect in the making of a software. Especially if its hosted in the internet, tight security is not an option. Research has been done as to which tool is the best in checking the vulnerability of a software. Le and Ray in their article on “Automated black box Web Application Vulnerability Testing”, put a lot of credit to black box scanners. They are very effective in identifying weaknesses. But are these black box scanners open sourced? No they are not. Some are open sourced to personal use only, like the Nessus scanner. Most of them are propitiated hence acquiring them is at a cost. Also maintenance cost comes in handy when you acquire them thus making the process expensive. Their services are impeccable but it’s at great cost.

Codes are prone to errors. Bugs present themselves in every stage of making a software. They are tedious and can delay the delivery process of a software. Developers have applied methods to take control of the issue but the processes have not come out triumphantly. Therefore, developers are required to come up with deigns that are going to be effective. Nevertheless, they are faced with some challenges in their designs. These challenges are: introduction of bugs in the code while trying to fix other bugs, the code to fix the bug is the bug itself, the design made to fix the bug will just cover up the worst and the implications to practitioners and researchers (Murphy, 2015). Therefore, choosing a design that will cover up all the niches is a great challenge. The black box method to analyze the output with reference to data sets and results that were predetermined can assist greatly in this field (Cox, 1999). The reference data sets and results are speculated according to the specification requirement of the software. Matching results of both the output of the software and the reference results proves that the software has achieved its objective. Though it uses speculation, this method has taken a great step in reducing the testing process. Also, an unskilled personnel in coding can test the software of its suitability to its functions.

Testing software can be done through four aspects (Kelly, 2011). These are: context, goals, adequacy and techniques. Each aspect or dimension as put by Kelly and Hook, are essential in the testing process and omitting one of them can lead to failure in the testing process. In addition to that, another aspect was introduced and that is, including a tester as an element in the testing process. By doing this, the tester is going to examine the code and combine with its execution. Furthermore, including the tester in the testing system will familiarize him with the code and ease the process of identifying bugs.

Challenges and Potential Solutions in Software Testing

Software testing can be a difficult and challenging. It is largely expensive, its outcome very unpredictable and excessively ad hoc (Bertolino, 2007). Hence, several scholars have tried to identify why it is so. Kanewala and Bienman in their article “Testing Scientific Software” identified these challenges. They pointed out that these difficulties occur due to this ad hoc challenge; cultural difference amongst softwares such as the oracle problem and distinguishing factors between scientist and the software development community. The oracle problem was also identified by Hook and Kelly in their article “Testing for trustworthiness in scientific software”.  They continued to state that a lot of test are required when conducting any testing mechanism. Considering the ad hoc challenge while choosing or coming up with a testing process is of great help since the problem can be dealt with accordingly. Software testing is process that most developers try to evade and do not put into place mechanism for testing to avoid last minute rushing and ensuring the software is working perfectly (Perry, 2007). Thus, if considered in time, like testing the code in portions, it would save so much time and money in the last test.

The testing method used in this research is the Gray method. I settled on this method since it has both the qualities of a black box and the white box methods. The tester does not require a deeper understanding of the inner working of the software and only requires some information about its inner functionalities. The best advantage of using this method is that it combines both the benefits of white box and black box methods. The sellAll program (the program under evaluation) is a program designed to sell goods and services online. I chose this program due to its vast uses like the Amazon online program. The program is easily understandable as its purpose is to display, sell and make delivery plans for the objects sold.  This program contains two major parts: the database and the user interface. The database must contain all the information about the products’ price, manufacturer, ingredients, types etc. This information is going to be used by the buyer to assess the product before purchasing it.

Several test could be performed on the program to check on its functionality. First, quality assurance must be made of the program. When the input is fed to the program, does the output satisfy the users expectations? This is a method of checking the quality of the software. A quality that is present in the blackbox method and in the gray box method. Next is the usability test. This involves including a tester externally to analyze the results of the program. The feedback from the tester will help determine if the user will be satisfied with the results and improvement done where it is required. This is a gray method too present in the blackbox method. Following the usability test is the interface test. Does the correct error message display when an error occurs? Is the system communication effective? These are questions the interface test should answer. A good user interface should alert the user on the latest product in the market, reduced price for some products, offers available and the like. Additionally, interruptions present should be taken care of appropriately. Another test that should be looked into with much care is the compatibility test. The program should be able to run in different browsers with the same level of responsiveness. Also, different versions of the browsers should not affect its operability. Another compatibility issue is the operating system. The program should not have problems when in different operating systems. The program will have to be run in different operating systems (OS) to rule out the compatibility with the OS. Testing for performance is crucial aspect too. How the program will respond under different internet speeds needs to be checked into. This will require the experts to check the quality of images to be included in the program. The images should have a clear resolution and pixel sizes should not be very high to allow fitting even in small screen devices. This will require the tester to analyze the pictures and the code. The quality should not be low and should be easily loaded under different data rates in the internet connection. The security of the software should be upheld with utmost care. Information about buyers’ credentials and buying details should not be disclosed to other buyers. Also information concerning the programs’ operability should be with the experts only and should not be disclosed under any circumstance. The test for this should be thorough and comprehensive.

The tester for this system only needs limited information of the program. This may just include interface information. The coding of the program is not too essential in the testing and the tester will just check if the program displays the products, sells the product and make delivery plans. Testing of the database will be paramount hence the presence of a database tester as a testing element in the testing system is of most importance. Information about the products will be fed into the system through the user interface in the administration panel and the tester will just check if the database is storing the information correctly. The information should also be viewed as it was stored and any information that was is not intended to be viewed by the customer should be well concealed.

I settled on TestNG since this testing framework tool has additional and easy to use testing functionalities over the JUNIT. TestNG has a wide range for testing, from unit testing to integration.  For security vulnerabilities, Yasca testing frame work saw it through. It is an open source hence acquiring it was not an issue. The program was examined by both frameworks and identified bugs were rectified immediately. Yasca also checked for code quality of the software hence making the programmers improve their codes to the standard ones. Selenium RC was introduced also since the program has a good reputation in testing and checking the quality of web applications. Loop holes not identified by the Yasca tool were identified using the Selenium RC program.

These three testing frameworks proved of great assistance in testing the program. Not only they helped in debugging it, they also helped in checking the performance of the program. For future programming, these testing tools are going to be used by my team. Not forgetting the gray testing method, the method that has made testing as simple as possible. Reducing the amount of work needed to test the whole code, hence saving on time and manpower. It is therefore preferable to use the grey method in testing since it combines both the benefits of the black box and white box methods.

Conclusions

Software testing is a broad area and should be done with a lot of keenness. Choosing an appropriate method and tool is of great importance too. For a software to be tested successfully, including a scientific tester as an element in the testing process is of essence, debugging the code while executing is far much better than waiting for the code to be done with, test it, then executing it. The gray testing method is better compared to the others since it combines both the advantages of the black and white methods. Web applications are best assessed with the Black box security scanners due to their agility in checking vulnerability in security in web applications. The SellAll online shopping application is a vast application that is nearly used worldwide. Its security vulnerabilities must be taken care of to avoid the issue of security breaches. Nevertheless, the application must undergo all the testing from its interface to its compatibility. Lastly, when fixing a bug, developers should come up with a testing technique that should not compromise the original code in any way. Bugs should not be introduced during the testing process.

References

Perry, W.E., 2007. “Effective methods for software testing: Includes complete guidelines, Checklists, and Templates.” John Wiley & Sons.

Banabic, R. and Candea, G., 2012, April. Fast black-box testing of system recovery code. In Proceedings of the 7th ACM european conference on Computer Systems (pp. 281-294). ACM.

Altaf, J.A. Dar, F.u. Rashid and M. Ra q, “Survey on selenium tool in software testing,” Green Computing and Internet of Things (ICGCIoT), 2015 International Conference on, Noida, 2015, pp.1378-1383.

Holmes and M. Kellogg, “Automating functional tests using Selenium,” AGILE 2006 (AGILE’ 06), Minneapolis, MN, 2006.

A.M.F.V. de Castro, G.A. Macedo, E.F. Collins and A.C. Dias-Neto, “Extension of Selenium RC tool to perform automated testing with databases in web applications,” Automation of Software Test (AST), 2013 8th International Workshop on, San Francisco, CA, 2013, pp. 125-131.

Radu Banabic and George Candea. 2012. “Fast Black-box testing of system recovery code.” In Proceedings of the 7th ACM European conference on Computer Systems (EuroSys ’12). ACM, New York, NY, USA, pp. 281-294.

Murphy-Hill, T. Zimmermann, C. Bird and N. Nagappan, “The Design Space of Bug Fixes and How Developers Navigate It,” in IEEE Transactions on Software Engineering, vol. 41, no. 1, 65-81, Jan. 1 2015.

Xu, W. Xu, B.K. Bavikati and W.E. Wong, “Mining Executable Specifications of Web Applications from Selenium IDE Tests,”, 2012 IEEE Sixth International Conference on Software Security and Reliability (SERE), Gaithersburg, MD, 2012, pp. 263-272.

Bau, E. Bursztein, D. Gupta and J. Mitchell, “State of the Art: Automated Black-Box Web Application Vulnerability Testing,” 2010 IEEE Symposium on Security and Privacy, Oakland, CA, USA, 2010, pp. 332-345.

Le and J.T. Ray eld, “Web-application development using the Model/View/Controller design pattern,” Proceedings. of the Fifth IEEE International Enterprise Distributed Object Computing Conference (EDOC’01), Seattle, WA, 2001, pp. 118-127.

Cox, M.G. and Harris, P.M., 1999. “Design and use of reference data sets for testing scientific software.” Analytica Chimica Acta, 380(2), pp.339-351.

Kelly, D., Thorsteinson, S. and Hook, D., 2011. “Scientific software testing: analysis with four dimensions.” IEEE software, 28(3), pp.84-90.

Kanewala, U. and Bieman, J.M., 2014. “Testing scientific software: A systematic literature review.” Information and software technology 56(10), pp. 1219-1232.

Hinsen, K., 2015. “The approximation tower in computational science: Why testing scientific software is difficult.” Computing in Science & Engineering, 17(4), pp. 72-77.  Bertolino, A., 2007, May. “Software testing research: Achievements, challenges, dreams.” In 2007 Future of Software Engineering (pp. 85-103). IEEE Computer Society.

Hook, D. and Kelly, D., 2009, May. “Testing for trustworthiness in scientific software.” In Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering (pp. 59-64). IEEE Computer Society.

Joppa, L.N., McInerny, G., Harper, R., Salido, L., Takeda, K., O’Hara, K., Gavaghan, D. and Emmott, S., 2013. “Troubling trends in scientific software use.” Science, 340(6134), pp. 814-815.

Romdhane, T., Software AG, 2015. White-box testing systems and/or methods for use in connection with graphical user interfaces. U.S. Patent 9,047,413.

 Kumar, M., Singh, S.K. and Dwivedi, R.K., 2015. A Comparative Study of Black Box Testing and White Box Testing Techniques. International Journal of Advance Research in Computer Science and Management Studies, 3(10).

 Jan, S.R., Shah, S.T.U., Johar, Z.U., Shah, Y. and Khan, F., 2016. An Innovative Approach to Investigate Various Software Testing Techniques and Strategies. International Journal of Scientific Research in Science, Engineering and Technology (IJSRSET), Print ISSN, pp.2395-1990.

 Wiener, J. and Calco, R., Wiener Jay Stuart and Calco Robert Becka, 2003. Method for merging white box and black box testing. U.S. Patent Application 09/946,282.

Yang, W., Prasad, M.R. and Xie, T., 2013, March. A grey-box approach for automated GUI-model generation of mobile applications. In International Conference on Fundamental Approaches to Software Engineering (pp. 250-265). Springer, Berlin, Heidelberg.