performance - How slow is too slow for unit tests? -


michael feathers, in working legacy code, on pages 13-14 mentions:

a unit test takes 1/10th of second run slow unit test... if [unit tests] don't run fast, aren't unit tests.

i can understand why 1/10th second slow if 1 has 30,000 tests, take close hour run. however, mean 1/11th of second better? no, not (as it's 5 minutes faster). hard fast rule isn't perfect.

thus when considering how slow slow unit tests, perhaps should rephrase question. how long long developer wait unit test suite complete?

to give example of test speeds. take @ several mstest unit test duration timings:

0.2637638 seconds 0.0589954 0.0272193 0.0209824 0.0199389 0.0088322 0.0033815 0.0028137 0.0027601 0.0008775 0.0008171 0.0007351 0.0007147 0.0005898 0.0004937 0.0004624 0.00045 0.0004397 0.0004385 0.0004376 0.0003329 

the average 21 of these unit tests comes 0.019785 seconds. note slowest test due using microsoft moles mock/isolate file system.

so example, if unit test suite grows 10,000 tests, could take on 3 minutes run.

i've looked @ 1 such project number of unit tests made system take long test everything. "too long" meaning didn't part of normal development routine.

however, had done categorize unit tests 2 parts. critical tests, , "everything else".

critical tests took few seconds run, , tested critical parts of system, "critical" here meant "if wrong here, everything going wrong".

tests made entire run take long relegated "everything else" section, , run on build server.

whenever committed code source control repository, critical tests again run first, , "full run" scheduled few minutes future. if nobody checked in code during interval, full tests run. granted, didn't take 30 minutes, more 8-10.

this done using teamcity, if 1 build agent busy full unit test suit, other build agents still pick normal commits , run critical unit tests needed.


Comments

Popular posts from this blog

unicode - Are email addresses allowed to contain non-alphanumeric characters? -

C#: Application without a window or taskbar item (background app) that can still use Console.WriteLine() -

c++ - Convert big endian to little endian when reading from a binary file -