This discussion is archived
1 2 Previous Next 18 Replies Latest reply: Nov 22, 2012 1:06 AM by TPD-Opitz-Consulting-com Go to original post RSS
  • 15. Re: JUnit: test that functions were performed in a specific order
    rp0428 Guru
    Currently Being Moderated
    >
    And how would you make such a 'universal' test pass?
    >
    I wouldn't - that was my point. A test that fails because you haven't written any code yet could be (but shouldn't be) testing anything at all.
    >
    The aim is always to make our production code pass all test at the end of the day.
    >
    Absolutely! But on my teams that is the goal for ALL projects in ALL environments, not just production. And that is one of the fundamental disagreements I have about the 'tests should fail initially and that is ok' perspective.

    We run nightly automated unit tests in all of our environments. Just in the DEV environment we are talking thousands of tests for an application that is being enhanced by 6-10 developers. The goal of the nightly tests is that they should all SUCCEED. The philosophy that it is ok for some tests to fail makes it impossible to know which failures are acceptable ('it is a new feature boss so it is supposed to fail') and which failures indicate an actual problem.

    Instead if every test (even on a mock or pseudo-code method) is required to succeed then it is very easy to know when you have a problem. The automated nightly process emails everyone on the team a detailed report that shows every test run and whether it succeeded or failed. Each team member is responsible for addressing the 'failures' for the areas assigned to them and if they haven't addressed them by the end of the day they had better have a very good reason such as a production issue taking their time.

    I can't imagine how you would hold people accountable for the work they do if they always have a bunch of tests that fail. I don't have time to investigate every failure to make sure it was 'ok' because the development wasn't done yet. There isn't anything attribute you can add to a test to designate which failure is 'ok' and which is a failure that should not have occured.

    That is the primary reason for my disagreement about writing tests that fail. If you require that all tests succeed then it becomes trivial to determine who is having problems and what area of the project the problems are occuring.

    Write the code for a feature or enhancement (mock, pseudo-code or actual modifications) and then write a new test or modify an existing one. If the code isn't ready to pass a test that code should not be hooked into the application yet. Each developer codes and tests individually in their own environment and copy of the app. The nightly process tests the ONE, OFFICIAL copy of the app in that environment. If a developer's modifications are not ready to be tested by the nightly processs they should not migrate those changes to the OFFICIAL copy of the app.

    We use this process to make sure that one developers work is not adversely affected by the work other developers may be doing. So in one sense how a developer works within their own environment is up to them. They can write 'failing' tests first if that is what they believe in or they can write mocks and pseudo-code first and then the test. I could care less. But at the point they add their changes to the 'official' app in any environment, including DEV, their tests had better SUCCEED or they will be held accountable. The daily test reports are an important part of their periodic performance reviews.

    I'm not trying to convince anyone that my perspective is right and theirs is wrong. But the approach we use accomplishes ALL of the objectives of TDD with only some minor 'bending' of the phraseology that is used. Present a failing test to one of your developers that is not involved with the project and I would expect them to have one heck of a time trying to determine if that failure is 'ok' or if the developer that did it is in over their head and needs help. With my approach it is very easy to get metrics that show who is having trouble.

    Two philosophies maybe but the goal is the same.
  • 16. Re: JUnit: test that functions were performed in a specific order
    TPD-Opitz-Consulting-com Expert
    Currently Being Moderated
    rp0428 wrote:
    I can't imagine how you would hold people accountable for the work they do if they always have a bunch of tests that fail.
    Now I see your problem.
    The point is: There is no 'bunch of tests that fail'. There is always one single test per developer that fails while she is working on the feature beeing tested. At the end of the day the feature should be compleeted and your nightly test run should be all green. At least the code change will not be committet to the SCM or CI unless this new test passes too.
    I guess the latter is what your developers do anyway. I don't think you'd accept untested unfinnished work in your SCM or CI environment.
    We use this process to make sure that one developers work is not adversely affected by the work other developers may be doing. So in one sense how a developer works within their own environment is up to them. They can write 'failing' tests first if that is what they believe in or they can write mocks and pseudo-code first and then the test. I could care less. But at the point they add their changes to the 'official' app in any environment, including DEV, their tests had better SUCCEED or they will be held accountable.
    So you see this from a higher level, not from the developers perspective. And from there I totally agree with you. Changes must not be committed unless all test pass.

    But eaven on this level:
    You surely have automated acceptance tests.
    Writing automated acceptance tests after the tested features are implemented gives you a count of compleeted features. But there is a (little) risk that you test the features that you implemented, not the features claimed by the requirements (sometimes this meigth be a difference... ;o)).
    Writing acceptance tests before the tested features are implemented gives me a count of what is still to do until my actual release is ready for shipping. And the test are be based on the requirements only ('cause there's nothing else they could be based on).

    But after all this is another pair of shoes...
    Present a failing test to one of your developers that is not involved with the project and I would expect them to have one heck of a time trying to determine if that failure is 'ok' or if the developer that did it is in over their head and needs help.
    This is verry true, the only other persons that should see the failing unit test is the pair-programming partner...
    With my approach it is very easy to get metrics that show who is having trouble.
    Would you share what that metric measures?

    bye
    TPD
  • 17. Re: JUnit: test that functions were performed in a specific order
    rp0428 Guru
    Currently Being Moderated
    >
    The point is: There is no 'bunch of tests that fail'. There is always one single test per developer that fails while she is working on the feature beeing tested. At the end of the day the feature should be compleeted and your nightly test run should be all green. At least the code change will not be committet to the SCM or CI unless this new test passes too.
    I guess the latter is what your developers do anyway.
    >
    Ah. . . and now I see your 'problem' (i.e. perspective). Even if there were one single test per developer for the feature our developers are typically assigned multiple features (e.g. bug fixes) and often work on multiple projects. Naturally the less experienced developers will work on fewer 'features' or projects at one time and are often paired with a more senior developer (which thrills them to no end, as you can imagine) to try to keep things a little more under control.
    >
    I don't think you'd accept untested unfinnished work in your SCM or CI environment.
    >
    That's the fantasy but not often the reality. More on that in the 'metrics' section.
    >
    So you see this from a higher level, not from the developers perspective
    >
    Yes - but on some projects I am also the technical lead or a senior developer for the more advanced/complex features so I see it from all levels.
    >
    You surely have automated acceptance tests.
    Writing automated acceptance tests after the tested features are implemented gives you a count of compleeted features. But there is a (little) risk that you test the features that you implemented, not the features claimed by the requirements (sometimes this meigth be a difference... ;o)).
    >
    Absolutely - each project has independent automated test runs.
    1. deletion of test database(s)
    2. restore of cold backup of test database (new cold backup test DB prepared after each official release)
    3. checkout from version control of the appropriate code tree (all using parameter script files to build the appropriate directory structures)
    4. complete rebuild of release code - if this step fails the developer(s) responsible for checking in the problem pieces get a serious talking to.
    5. automated unit test run - test results are output to dedicated test folders created during step #3 and #4.
    6. automated email of html test results to all members of the project. The reports include a summary section showing the feature, # tests run, # failed, #serious failures. The detail section includes a line for every test with the same info and includes detailed info for that test about the parameters used.

    At the first daily scrum meeting (yep - agile) every developer reports on their test failures and when they will be fixed. It is best not to have any. ;)
    >
    Writing acceptance tests before the tested features are implemented gives me a count of what is still to do until my actual release is ready for shipping. And the test are be based on the requirements only ('cause there's nothing else they could be based on).
    >
    Using Agile methodology project progress, including feature and test status and progress is managed daily and updates given in the scrum meetings based on the project workflow documentation. Each developer reports: what did you get done yesterday? what are you doing today? Do you have any 'impediments'? Test failures are 'front of the line' impediments.
    >
    Would you share what that metric measures?
    >
    The 'planning poker' phase of the agile process assigns a relative 'complexity' or 'ease of development' to each feature, or user story, of the project. (see Planning Poker - http://en.wikipedia.org/wiki/Planning_poker).

    So using 'simple', 'moderate', 'complex' you have a list of developers. For each developer there is a list of their projects. For each project a list of 'features' (bug fixes, enhancements, etc). Each feature has a complexity and a timeline (including the sprints they are part of - http://en.wikipedia.org/wiki/Scrum_(development)) in the project.

    The metrics include the number and type (simple, etc) of tests completed and failed and the numbers of met/missed sprint deadlines per developer. So developers have an incentive to not take on more work than they can actually do and to not take on work that is more complex than they can handle. You can't just take on work, you have to get it done according to the schedule or it gets noticed.

    Unfortunately with agile/scrum it is easy for developers to volunteer to work on things they aren't really qualified to do. But at least the nightly automated tests and project reporting if you take on a feature that you aren't really qualified to work on it shows up pretty quick when your required tests are either missing from the nightly run, have failed during the run or your code changes cause other developer's test to fail because you checked in changes that still had problems.

    The metrics show which developers repeatedly have failed tests and whether those failures are for 'simple', 'moderate' or 'complex' features. A failure on a 'simple', fairly isolated bug fix even after a few days may not be significant; the bug fix can possibly be delayed until the next sprint or release. But a test failure in a complex feature gets immediate attention from the team and immediate help for the developer if needed.

    Hope that helps explain the type of 'metrics' I was talking about.
  • 18. Re: JUnit: test that functions were performed in a specific order
    TPD-Opitz-Consulting-com Expert
    Currently Being Moderated
    rp0428 wrote:
    [...] our developers are typically assigned multiple features (e.g. bug fixes) and often work on multiple projects.
    Do they also work on multiple features and projects at the same time? Does a developer switch between assigned taskt every 2 muntes?
    Hopefully she will pick one of the assigned taskes an finnishes this before switching to the next. And when "beeing finnished" means "production code +and+ test completed" then it does not mater when the test was written. Your central codeebase will never see a failing unit test.
    But you already committed to that. And if I got you right you allow your developers to work that way, just don't force TDD.
    >
    I don't think you'd accept untested unfinnished work in your SCM or CI environment.
    >
    That's the fantasy but not often the reality.
    How do you detect unfinnished (hence untested) code in you codebase?
    I guess via coverage analysis, right?
    At the first daily scrum meeting (yep - agile) every developer reports on their test failures and when they will be fixed. It is best not to have any. ;)
    Ah, the strech bench next to the coffemaker... ;o)
    But you stated that the test your developers write work on existing code and they therefore never fail.
    Are the failing test ususally "collateral damages" to preexisting code?
    Or do you have separate developers writing test for the code written bye somone else?

    And what is the "trigger" for your developers to commit to the SCM? Each day at 4PM? Every 2 hours?
    In the TDD cycle I commit each work package twice: when the current (and only one failing) test turned green and when I finnished refactoring.
    So my central codebase does never have failing unit test. This leads to a "unsteadieness" in my commits, but I think it's a fair tradeof for a "always green" central codebase.
    [more on the metrics]
    Hope that helps explain the type of 'metrics' I was talking about.
    Yes.
    Thank you.

    bye
    TPD
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points