U ovom ćemo članku naučiti jedinstveni i učinkoviti pristup ispitivanju koji se naziva ispitivanje temeljeno na svojstvima. Za provedbu ovog pristupa testiranju koristit ćemo Python , pytest i Hypothesis .
Članak će koristiti osnovne koncepte pytesta kako bi objasnio ispitivanje temeljeno na svojstvima. Preporučujem vam da pročitate ovaj članak kako biste brzo očistili svoje pytest znanje.
Započet ćemo s konvencionalnom metodom jedinstvenog / funkcionalnog ispitivanja poznatom kao testiranje na primjeru -koje većina nas koristi. Pokušavamo pronaći njegove nedostatke, a zatim prelazimo na pristup temeljen na imovini kako bismo uklonili te nedostatke.
Svaki veliki magični trik sastoji se od tri dijela ili radnje. Prvi dio naziva se "Zalog". Mađioničar vam pokazuje nešto obično : špil karata, ptica ili čovjek. Pokazuje vam ovaj predmet. Možda od vas traži da ga pregledate kako bi provjerio je li doista stvaran, nepromijenjen i normalan. Ali naravno ... vjerojatno nije.
1. dio: Ispitivanje na primjeru
Pristup testiranju na primjeru ima sljedeće korake:
- s obzirom na testni unos I
- kada se prenese na funkciju koja se ispituje
- treba vratiti izlazni O
Dakle, u osnovi dajemo fiksni ulaz i očekujemo fiksni izlaz.
Da bismo laički razumjeli ovaj koncept:
Pretpostavimo da imamo stroj koji za ulaz uzima plastiku bilo kojeg oblika i bilo koje boje i proizvodi savršeno okruglu plastičnu kuglu iste boje kao i izlaz.
Sada ćemo, kako bismo testirali ovaj stroj pomoću primjera zasnovanog testiranja, slijediti pristup u nastavku:
- uzeti sirovu plastiku plave boje ( fiksni podaci o ispitivanju )
- umetnite plastiku u stroj
- očekujte plastičnu kuglu plave boje kao izlaz ( fiksni testni izlaz )
Pogledajmo isti pristup na programski način.
Preduvjet: provjerite imate li instaliran Python (verziju 2.7 ili noviju) i pytest .
Stvorite strukturu direktorija poput ove:
- demo_tests/ - test_example.pyU sumdatoteku ćemo napisati jednu malu funkciju test_example.py. Ovo prihvaća dva broja - num1i num2 - kao parametre i vraća zbrajanje oba broja kao rezultat.
def sum(num1, num2): """It returns sum of two numbers""" return num1 + num2Sada, napišite test za testiranje ove funkcije zbroja prema uobičajenoj metodi.
import pytest#make sure to start function name with testdef test_sum(): assert sum(1, 2) == 3Ovdje možete vidjeti kako smo se donošenje dvije vrijednosti 1i 2, a očekuje se iznos za povrat 3.
Pokrenite testove prelaskom u demo_testsmapu, a zatim izvođenjem sljedeće naredbe:
pytest test_example.py -v
Je li ovaj test dovoljan za provjeru funkcionalnosti sumfunkcije?
Možda razmišljate, naravno da ne. Napisat ćemo još testova koristeći pytest parametrizeznačajku koja će izvršiti ovu test_sumfunkciju za sve zadane vrijednosti.
import pytest@pytest.mark.parametrize('num1, num2, expected',[(3,5,8), (-2,-2,-4), (-1,5,4), (3,-5,-2), (0,5,5)])def test_sum(num1, num2, expected): assert sum(num1, num2) == expected
Korištenje pet testova dalo je više samopouzdanja u funkcionalnost. Svi oni koji prolaze osjećaju se kao blaženstvo.
Ali , ako pažljivije pogledate, radimo isto što i gore, ali radi većeg broja vrijednosti. Još uvijek ne pokrivamo nekoliko rubnih slučajeva.
Dakle, otkrili smo prvu bolnu točku ovom metodom ispitivanja:
Pitanje 1: Iscrpnost testa ovisi o osobi koja je napisala test
Oni mogu odlučiti napisati 5 ili 50 ili 500 testnih slučajeva, ali još uvijek nisu sigurni jesu li sigurno pokrili većinu, ako ne i sve rubne slučajeve.
To nas dovodi do naše druge bolne točke:
2. izdanje - Nerobusni testovi zbog nejasnog / dvosmislenog razumijevanja zahtjeva
Kad su nam rekli da napišemo svoju sumfunkciju, koji su se konkretni detalji prenijeli?
Jesu li nam rekli:
- kakav unos naša funkcija treba očekivati?
- kako bi se naša funkcija trebala ponašati u neočekivanim ulaznim scenarijima?
- kakav bi izlaz naša funkcija trebala vratiti?
Točnije, ako uzmete u obzir sumfunkciju koju smo gore napisali:
- znamo li
num1,num2treba li bitiintilifloat? Mogu li se također poslati kao vrstastringili bilo koja druga vrsta podataka? - what is the minimum and maximum value of
num1andnum2that we should support? - how should the function behave if we get
nullinputs? - should the output returned by the sum function be
intorfloatorstringor any other data type? - in what scenarios should it display error messages?
Also, the worst case scenario of the above test case writing approach is that these test cases can be fooled to pass by buggy functions.
Let’s re-write our sum function in a way that errors are introduced but the tests which we have written so far still passes.
def sum(num1, num2): """Buggy logic""" if num1 == 3 and num2 == 5: return 8 elif num1 == -2 and num2 == -2 : return -4 elif num1 == -1 and num2 == 5 : return 4 elif num1 == 3 and num2 == -5: return -2 elif num1 == 0 and num2 == 5: return 5
Now let’s dive into property-based testing to see how these pain points are mitigated there.
Drugi čin naziva se "Zaokret". Mađioničar uzima nešto obično i tjera ga da čini nešto izvanredno. Sada tražite tajnu ... ali nećete je naći, jer naravno da zapravo ne tražite. Ne želite zapravo znati. Želiš se prevariti.
Dio 2: Ispitivanje temeljeno na svojstvima
Uvod i generiranje test podataka
Ispitivanje temeljeno na svojstvima prvi je put uveo okvir QuickCheck u Haskellu . Prema dokumentaciji brze provjere, koja je još jedna biblioteka za testiranje temeljena na svojstvima -
Okviri ispitivanja temeljeni na svojstvima provjeravaju istinitost svojstava. Svojstvo je izjava poput: za sve (x, y, ...) kao što je preduvjet (x, y, ...) vrijedi svojstvo (x, y, ...) vrijedi .To understand this let’s go back to our plastic ball generating machine example.
The property based testing approach of that machine will be:
- take a huge selection of plastics as input (
all(x, y, …)) - make sure all of them are colored (
precondition(x, y, …)) - the output satisfies following property (
property(x, y, …)) -
- output is round/spherical in shape
- output is colored
- color of the output is one of the colors present in color band
Notice how from fixed values of input and output we have generalized our test data and output in such a way that the property should hold true for all the valid inputs. This is property-based testing.
Also, notice that when thinking in terms of properties we have to think harder and in a different way. Like when we came up with the idea that since our output is a ball it should be round in shape, another question will strike you - whether the ball should be hollow or solid?
So, by making us think harder and question more about the requirement, the property-based testing approach is making our implementation of the requirement robust.
Now, let’s return to our sum function and test it by using the property-based approach.
The first question which arises here is: what should be the input of the sum function?
For the scope of this article we will assume that any pair of integers from the integer set is a valid input.
So, any set of integer values lying in the above coordinate system will be a valid input to our function.
The next question is: how to get such input data?
The answer to this is: a property-based testing library provides you the feature to generate huge set of desired input data following a precondition.
In Python, Hypothesis is a property-testing library which allows you to write tests along with pytest. We are going to make use of this library.
The entire documentation of Hypothesis is beautifully written and available ➡️ hereand I recommend you to go through it.
To install Hypothesis:
pip install hypothesisand we are good to use hypothesis with pytest.
Now, let’s rewrite test_sum function — which we wrote earlier — with new data sets generated by Hypothesis.
from hypothesis import givenimport hypothesis.strategies as stimport pytest@given(st.integers(), st.integers())def test_sum(num1, num2): assert sum(num1, num2) == num1 + num2- The first line simply imports
givenfrom Hypothesis. The@givendecorator takes our test function and turns it into a parametrized one. When called, this will run the test function over a wide range of matching data. This is the main entry point to Hypothesis. - The second line imports
strategiesfrom Hypothesis. strategies provides the feature to generate test data. Hypothesis provides strategies for most built-in types with arguments to constrain or adjust the output. As well, higher-order strategies can be composed to generate more complex types. - You can generate any or mix of the following things using strategies:
'nothing','just', 'one_of','none','choices', 'streaming','booleans', 'integers', 'floats', 'complex_numbers', 'fractions','decimals','characters', 'text', 'from_regex', 'binary', 'uuids','tuples', 'lists', 'sets', 'frozensets', 'iterables','dictionaries', 'fixed_dictionaries','sampled_from', 'permutations','datetimes', 'dates', 'times', 'timedeltas','builds','randoms', 'random_module','recursive', 'composite','shared', 'runner', 'data','deferred','from_type', 'register_type_strategy', 'emails'- Here we have generated
integers()set using strategies and passed it to@given. - So, our
test_sumfunction should run for all the iterations of given input.
Let’s run it and see the result.
You might be thinking, I can’t see any difference here. What’s so special about this run?
Well, to see the magical difference, we need to run our test by setting the verbose option. Don’t confuse this verbose with the -v option of pytest.
from hypothesis import given, settings, Verbosityimport hypothesis.strategies as stimport pytest@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())def test_sum(num1, num2): assert sum(num1, num2) == num1 + num2settings allows us to tweak the default test behavior of Hypothesis.
Now let’s re-run the tests. Also include -s this time to capture the stream output in pytest.
pytest test_example.py -v -s
Look at the sheer number of test-cases generated and run. You can find all sorts of cases here, such as 0, large numbers, and negative numbers.
You might be thinking, it’s impressive, but I can’t find my favorite test case pair (1,2 ) here. What if I want that to run?
Well, fear not, Hypothesis allows you to run a given set of test cases every time if you want by using the @example decorator.
from hypothesis import given, settings, Verbosity, exampleimport hypothesis.strategies as stimport pytest@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())@example(1, 2)def test_sum(num1, num2): assert sum(num1, num2) == num1 + num2
Also, notice that each run will always generate a new jumbled up test case following the test generation strategy, thus randomizing the test run.
So, this solves our first pain point- the exhaustiveness of test cases.
Thinking hard to come up with properties to test
So far, we saw one magic of property-based testing which generates desired test data on the fly.
Now let’s come to the part where we need to think hard and in a different way to create such tests which are valid for all test inputs but unique to sum function.
1 + 0 = 10 + 1 = 15 + 0 = 5-3 + 0 = -38.5 + 0 = 8.5Well, that’s interesting. It seems like adding 0 to a number results in the same number as sum. This is called the identity property of addition.
Let’s see one more:
2 + 3 = 53 + 2 = 55 + (-2) = 3-2 + 5 = 3It looks like we found one more unique property. In addition the order of parameters doesn’t matter. Placed left or right of the + sign they give the same result. This is called the commutative property of addition.
There is one more, but I want you to come up with it.
Now, we will re-write our test_sum to test these properties:
from hypothesis import given, settings, Verbosityimport hypothesis.strategies as stimport pytest@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())def test_sum(num1, num2): assert sum(num1, num2) == num1 + num2 # Test Identity property assert sum(num1, 0) = num1 #Test Commutative property assert sum(num1, num2) == sum(num2, num1)
Naš je test sada iscrpan - pretvorili smo i testove kako bismo ih učinili robusnijima. Tako smo riješili našu drugu bolnu točku: ne-robusni test slučajevi .
Samo iz znatiželje, pokušajmo zavarati ovaj test onim buggy kodom koji smo prije koristili.
Kao što kaže stara poslovica - prevarite me jednom, sramite se, prevarite me dvaput, sramite me.
Možete vidjeti da je uhvatila pogrešku. Falsifying example: test_sum(num1=0, num2=0). To jednostavno znači da naše očekivano svojstvo nije vrijedilo za ove parove testnih slučajeva, a time i neuspjeh.
Ali još ne biste pljeskali. Jer natjerati nešto da nestane nije dovoljno; moraš ga vratiti. Zato svaki magični trik ima treći čin, najteži dio, dio koji nazivamo "Prestiž".
Dio 3: Skupljanje neuspjeha
Shrinking is the process by which Hypothesis tries to produce human-readable examples when it finds a failure. It takes a complex example and turns it into a simpler one.
To demonstrate this feature, let’s add one more property to our test_sum function which says num1 should be less than or equal to 30.
from hypothesis import given, settings, Verbosityimport hypothesis.strategies as stimport pytest@settings(verbosity=Verbosity.verbose)@given(st.integers(), st.integers())def test_sum(num1, num2): assert sum(num1, num2) == num1 + num2 # Test Identity property assert sum(num1, 0) = num1 #Test Commutative property assert sum(num1, num2) == sum(num2, num1) assert num1 <= 30After running this test, you will get an interesting output log on the terminal here:
collected 1 itemtest_example.py::test_sum Trying example: test_sum(num1=0, num2=-1)Trying example: test_sum(num1=0, num2=-1)Trying example: test_sum(num1=0, num2=-29696)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=-1763, num2=47)Trying example: test_sum(num1=6, num2=1561)Trying example: test_sum(num1=-24900, num2=-29635)Trying example: test_sum(num1=-13783, num2=-20393)#Till now all test cases passed but the next one will failTrying example: test_sum(num1=20251, num2=-10886)assert num1 <= 30AssertionError: assert 20251 <= 30#Now the shrinking feature kicks in and it will try to find the simplest value for which the test still failsTrying example: test_sum(num1=0, num2=-2)Trying example: test_sum(num1=0, num2=-1022)Trying example: test_sum(num1=-165, num2=-29724)Trying example: test_sum(num1=-14373, num2=-29724)Trying example: test_sum(num1=-8421504, num2=-8421376)Trying example: test_sum(num1=155, num2=-10886)assert num1 <= 30AssertionError: assert 155 <= 30# So far it has narrowed it down to 155Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=64, num2=0)assert num1 <= 30AssertionError: assert 64 <= 30# Down to 64Trying example: test_sum(num1=-30, num2=0)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=0, num2=0)Trying example: test_sum(num1=31, num2=0)# Down to 31Trying example: test_sum(num1=-30, num2=0)Falsifying example: test_sum(num1=31, num2=0)FAILED# And it finally concludes (num1=31, num2=0) is the simplest test data for which our property doesn't hold true.
One more good feature — its going to remember this failure for this test and will include this particular test case set in the future runs to make sure that the same regression doesn’t creep in.
This was a gentle introduction to the magic of property based testing. I recommend all of you try this approach in your day to day testing. Almost all major programming languages have property based testing support.
You can find the entire code used here in my ? github repo.
If you liked the content show some ❤️