Service Virtualization

Expand all | Collapse all

Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

  • 1.  Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Posted Dec 15, 2017 01:26 AM

    I ran a test suite which has 3 test cases(ran locally) . I thought my assertions in each test case got passed. So again I tried out in another way. I manually made an assertion in one of the test case to be "false"(by giving some wrong value). This time when I ran the suite again (ran locally) and got the same result which is "PASS" for 3 test cases. I dont know what happened here. Even my assertion is false , how the test case got passed.? Whats the point of automating this? If the API doesnt give response , will my test case pass too?

     

    Is there a place where I can see the results for each assertion whether it got passed or failed because I can see only the result for entire test case.

     

    case 1: I gave like if my assertion is "true" then "end the test". In this case my assertion got succeeded (because i gave correct value in "expected value" field) and all my test cases got passed.

     

    case 2: I gave like if my assertion is "true" then end the test.In this case my assertion got failed (because I gave a wrong value in "expected value" field). This time also all my 3 test cases got passed.

     

    Note: assertion is in my 1st test case(out of three test cases)

     

    If the service which I m testing is not working, then case2 will happen, correct?.So in that case, how will I know if my test case got passed?

     My question is : I am expecting a value "x" in one field which I will keep it as "expected value". So, if my service has no issues then "x" will be obtained in the response and then it will match exactly with my "expected value". What if when my actual service is down? I wont get "x" in my response. Then my assertion will become false. If my assertion is false....my service response is wrong.....and if it is wrong, how can my test case get passed?Is there anyother way my test case (only one case out of three) get failed if my actual service is down. Or is there any other option to do my negative scenario? 

     

    And I m not sure all of my assertions are executed if I have multiple assertions in my test case. 

     

     

    see attached screen shot

    Attachment(s)



  • 2.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Broadcom Employee
    Posted Dec 17, 2017 11:59 PM

    Hi,

     

    If you define an assertion with the condition like:
    "If True then End the Test"
    then the test case ends only when the condition is "True".
    If the condition is "False", then the test case will go to the next step.

     

    To see the Screenshot for Case 2 in your word documents, the test case will go to the next step with negative scenario, because this assertion does not include the negative condition.
    You mentioned :
    I manually made an assertion in one of the test case to be "false"(by giving some wrong value).
    But this change does not mean that the tesst case will become "Failed".
    (This change only means that the "If" condition of the assertion is always "false".)

     

    If you want to add negative scenario like:
    "If False then Failed the Test"
    then you need to add an assertion which includes the negative scenario like below:


    Cheers



  • 3.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Posted Dec 18, 2017 01:09 AM

    okaye, so, if I have to fail with "false" I have to change to "fail the test". Fine. Now my questions are........

     

    1. In one test case, if I need to add 2 or more assertions which is going to be true (as my response will have 2 or more values) what should I do. How will dev test go to my second assertion if my first assertion is true?

     

    2. My test case will be like, if assertion is true then my test case should get passed and my assertion is false my test case should get failed. How can I give these both scenarios in as single test case? (in a single assertion or 2 assertion)



  • 4.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Broadcom Employee
    Posted Dec 18, 2017 01:44 AM

    Hi,

     

    1. In one test case, if I need to add 2 or more assertions which is going to be true (as my response will have 2 or more values) what should I do. How will dev test go to my second assertion if my first assertion is true?

     

    Ans. Do you need to add 2 or more assertions in one step or in one test case? If you need to add 2 or more assertions in one test step and the conditions in the assertions are related (if you want to create complex conditions with assertions), then you will need to create a "Scripted Assertion" and write a complex logic depends on the conditions.
    But if your test case includes multiple steps, the test case includes the multiple assertions but the each test case includes only one assertion(but this is based on your requirement), then you do not have to write a logic with a "Scripted Assertion".

     


    2. My test case will be like, if assertion is true then my test case should get passed and my assertion is false my test case should get failed. How can I give these both scenarios in as single test case? (in a single assertion or 2 assertion)

     

    Ans. That depends on your requirement.
    Here is an example:
    If I define one assertion which includes "If False then Failed the Test", then the assertion works like:
    a. if the condition becomes "True" then goes to next step
    b. if the condition becomes "False" then the status of the test case becomes "Failed" and the test case stops
    If the pattern a is OK for you, then you need only one assertion in your test step.

     

     

    Cheers,



  • 5.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Posted Dec 18, 2017 03:08 AM

    Hi,

     

    1. Actually this is not complex. I need 2 or more assertion in a single test case TC_1.Consider the below json is my response of my service "X".

     

    "johny": {         "subject": "maths",
            "mark": 100,
            "ispresent": true
           }
    Now I need to  verify all the three values. So i used a REST step for my service "X" and I got the above response. I inserted my first assertion like
    JSON path = $.johny.subject and expected value = "maths" and I gave the condition as "If 'true' end the test". So my first assertion will get passed and not go to second assertion.
    Is there any way to go to my second and third assertion and pass them? Is this the case where you are asking me to use scripted assertion?

    2. Check if I am correct.Forget about the first case.In the second case TC_2 ,I have to see two things. consider my REST service  "X" has some issues and
    retured some 500 error. Now I will give like "If 'False' then Fail the Test". So if I get 500 error,
    my test case will get failed which is correct. But in the case If my service returned correct response say 200,
    I will have my second assertion like
    "If 'true' then End the Test".

    So, if I get 200 it will got to my second case and pass the test.
    If I get 500 it will be failed in the first assertion itself.


  • 6.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Broadcom Employee
    Posted Dec 18, 2017 04:01 AM

    Hi,

     

    1. If you create an assertion with "If 'true' end the test" as first assertion, then the second or third assetions will be ignored when the condition is matched to the condition in the first assetion because you already defined the "end the test". 
       If you create an assertion with "If 'false' then something" and when the condition is matched to the condition in the first aseertion, then second assertion will check the condition.

       The process in the three assertions are like this:

       Condition for the first assertion -> is matched then do the description(like "End the Test") and the second, third assertions are ignored
            | is not matched
           V
       Condition for the second assertion ->  is matched then do the description(like "End the Test") and the third assertion is ignored
            | is not matched
           V
       Condition for the third assertion ->  is matched then do the description(like "End the Test")
            | is not matched
           V
       Go to the next step

       Then please be care for defining the description in the assertion definitions.

     

    2. I think you are correct.

     

    But please be careful if you define the multiple assertions in one step.
    Please look at the screenshot for "Execute Script(JSR-223)" step below, in this step the assertions will be processed like:
      a. If environment error is True then Abort the test. If environment error is False then go to the next assertion(Any Exception)
      b. If Any Exception is True then Fail the test. If Any Exception is False then go to the next assertion(Ensure Property Matches Expression)
      c. If Ensure Property Matches Expression is True then Fail the test. If Ensure Property Matches Expression is False then go to the next step

     

    Cheers,



  • 7.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Posted Dec 18, 2017 04:37 AM

    Hi,

     

    1. So what if I have to verify 2 or more node values from my output json?

     

    2. I don't have these type of assertions as my output is Json. 



  • 8.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?
    Best Answer

    Posted Dec 18, 2017 09:52 AM

    Firstly, Assertions are treated as booleans, so a test can encounter issues when the PASS/FAIL decision is formed on the basis of evaluating multiple conditions.

    When I run into situations where a test case has to deal with multiple assertions, I consider implementing an approach that includes script logic so the test can make a more comprehensive pass/fail decision.

     

    Consider the following example:

    1) Initialize an error message property or multiple properties. During the test, as checks "fail" add information to these properties. 

    2) Flow through the test case, as errors occur concatenate error information in these properties. This allows the test to create a list of all the errors encountered during the test rather than just Failing the test when the first error is encountered. 

    3) When the test case flow ends, add a script step that evaluates the overall condition of the test and either passes or fails. In the Pass/Fail test case script, evaluate the properties. If the property is empty (""), return PASS. If the property contains error information, return "FAIL". Add an assertion to this step that ends the test if the script's LASTRESPONSE is PASS and fails the test case if the LASTRESPONSE is FAIL. The test can write the error output to the file system if necessary. The file contains a description of all of the various errors that were encountered during the test which can help in debugging and minimize overall test time.

     

    Following is a fictitious example flow. The idea is that all of the potential error conditions are evaluated in a pass/fail step and the "list" of error conditions are appended to a failure report. Some of the errors are hard errors and end the test immediately (e.g., input comparison data is not found, or data is missing on the system of record).   

     



  • 9.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Posted Jan 18, 2018 07:29 AM

    Since I m a beginer to ca dev test, can u please let me know a small scripted assertion (consider the below response which has 2 values for which I have to verify both the values, are the ones which I expected). I am asking because I dono where the properties are and how to extract, those kind of things. I just know the reponse will be stored in a name "test.rsp" (may be a javascript or groovy)

    {

    "node ":  

    {

    "value1": "lisa one",'

    "value2": "lisa two"

                        }

    }



  • 10.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Posted Jan 18, 2018 09:52 AM
      |   view attached

    Refer to this guide for help with Scripting DevTest 8.0 - Scripting Guide - V1.1.pdf  

     

    Attached is one example of looping over a series of data and failing the test case one time if any of the data is invalid. This test case uses properties to store off information. When the last row of data is processed, a final step determines if the Test Case passes or fails. Again, this is just one approach but it demonstrates how several assertions can be validated in a single pass. Consider that the output of the test run with messages could be saved to the file system for research by a developer or tester.

    Execute this in ITR mode and single step through the test to watch what happens as different values pass and fail.

    Attachment(s)

    zip
    Validation Loop.tst.zip   2 KB 1 version


  • 11.  Re: Will all my assertions in my test case will run in CA dev test and how to do a negative scenario?

    Posted Feb 06, 2018 12:17 PM
      |   view attached

    Hi,

     

    Thanks alot for these documents. In this I have a query. 

     

    I created a test case which executes a web service. In the response xml, I used filters and stored 2 of the response values(see attached screen shot) so that I can use these values from "available objects" section during scripted assertion. Then I closed this test case and opened again. When I navigated to filter section, I see the previously filtered values are not highlighted in yellow colour. When I hit "run filter" button, I get blank. Please help me with this. Ho to store these values in object.

     

    See attached.

    Attachment(s)