env.do_test vs ovm_env::run_test and Report Summary

[Gareth]

Hi all,

I'm in the process of moving an existing testbench from AVM-style testcases to OVM factory produced testcases and there have been several hurdles on the way.

One thing I have noticed that appears on the face of it to be a bug is in the difference between the AVM-style env.do_test() call and the static ovm_env::run_test() call and it appears the examples show the same behaviour.

If you run ovm-1.0.1/examples/ovm_examples/tlm/fifo [which uses env.do_test()], at the end of the transcript you get a report summary:

# --- OVM Report Summary ---
# 
# ** Report counts by severity
# OVM_INFO :   20
# OVM_WARNING :    0
# OVM_ERROR :    0
# OVM_FATAL :    0
# ** Report counts by id
# [consumer            ]    10
# [producer            ]    10
# ** Note: $finish    : fifo.sv(132)
#    Time: 100 ns  Iteration: 10  Instance: /top

but in the ovm-1.0.1/examples/factory example [which uses ovm_env::run_test()], there is no report summary:

# OVM_INFO @ 75: ovm_test_top [PKTGEN] Got packet: p: (mypacket) 
#   addr: 'h9
#   data: 'he
# 
# ** Note: $finish    : ../../src/base/ovm_env.sv(274)
#    Time: 75 ns  Iteration: 11  Instance: /ovm_pkg::ovm_env::run_test

I'm about to start digging through the source to figure out what is going on; I also haven't yet investigated if any of the phases after run() are being executed.

In the meantime, is this intended behaviour? do run_test()-based testcases need something extra to be done to get them to report properly at the end of the simulation?

[anantv]

Hi

I am also facing the same issue:
After run_test() task call, there NO "OVM Report Summary" generated/displayed.

Whether the user has to manually call the report_summarize() task in his env class after run_test() call?

Expecting reply asap.

Thanks in advance.

TVAR

[kurts]

in AVM (and in OVM),

env_handle.do_test() essentially does the following:

print_copyright()
run_global_phases()
report_summarize()


run_test, however, does not do either the print copyright or the report_summarize. It basically just does the run_global_phases. I'm pretty sure that this is intended behavior. Now, it does do some more things, like automatically instantiate a component as a top-level test (using the factory), but thats it.

So if you want to get a report summary at the end of simulation, you should put a call to report_summarize() in the report() function of your top level (or your environment, if it is not the top level). That way, you will get a summary in the last phase after the run phase completes.

Now, I have seen what I consider to be a bug in the behavior of run_test. If you have *any* processes still "alive", even if they are currently dormant, blocking on an event, then when the environment run() function completes, no further phases are executed. That means that even if you have a report_summarize in your report method, it will not be called.

The workaround to this is to make a call to global_stop_request() (usually I put this at the end of my environment run() function). This seems to kill off all child processes and allows the remaning phases (extract, check, report) to execute.

-Kurt

[anantv]

Hi Kurts,

Thanks for detailed description.

I have added calls to report_header & report_summarize task in the report function of my top level env class.

Now I was able to get report summary at the end of the simulation in the run_test based test cases.

run_test, however, does not do either the print copyright or the report_summarize. It basically just does the run_global_phases. I'm pretty sure that this is intended behavior. Now, it does do some more things, like automatically instantiate a component as a top-level test (using the factory), but thats it.

Could you please brief on this intended behavior of the OVM on reporting???

Why can't those report calls be included by the library ? Is it related to running multiple run_test() with different env's??

Where does the static bit finish_on_completion in the ovm_env.svh used in these multiple run_test()?

By the way, am taking Tim classes on OVM at present.

Thanks in advance.


TVAR

[dlong]

Hi Kurt,

Now, I have seen what I consider to be a bug in the behavior of run_test. If you have *any* processes still "alive", even if they are currently dormant, blocking on an event, then when the environment run() function completes, no further phases are executed. That means that even if you have a report_summarize in your report method, it will not be called.

The workaround to this is to make a call to global_stop_request() (usually I put this at the end of my environment run() function). This seems to kill off all child processes and allows the remaning phases (extract, check, report) to execute.

The issue with run_test() appears to be that it doesn't set ovm_component::m_do_test_mode to 1 (unlike do_test()). I don't know if this is intentional or an oversight (perhaps one of the OVM developer's could shed some light on this?).

A simple workaround is to set this static member yourself. The simulation will then stop automatically at the end of your top-level env run task without the need to call global_stop_request. The configure method of your test class seems like a good place to do it, e.g.

class my_test extends ovm_test;

   verif_env env1;
               
   function new (string name, ovm_component parent);
      super.new(name,parent);
   endfunction : new

   virtual function void build();
      super.build();
      $cast(env1,ovm_factory::create_component("verif_env","","env1",null) );

  endfunction : build

   function void configure();
      ovm_component::m_do_test_mode = 1; 
   endfunction: configure

...
   
  `ovm_component_utils(my_test)

endclass: my_test

Note that you need to use ovm_factory::create_component rather than the test's create_component method since the environment's parent must be set to null rather than "this" (otherwise it won't be recognised as a top-level environment).

Regards,
Dave

[kurts]

Hi Dave,

Thanks for the workaround.

The issue with run_test() appears to be that it doesn't set ovm_component::m_do_test_mode to 1 (unlike do_test()). I don't know if this is intentional or an oversight (perhaps one of the OVM developer's could shed some light on this?).

Now that you mention it, I recall having a discussion a while back with some of the OVM developers where they warned me about "different ending semantics" between run_test() and do_test(). At the time, I wasn't sure what they meant, but this may be it. That leads me to believe that this is intended behavior, in order to preserve backward compatibility with both URM and AVM users. I agree that it would be good to get clarification from the OVM developers about this.

Could you please brief on this intended behavior of the OVM on reporting???

Why can't those report calls be included by the library ? Is it related to running multiple run_test() with different env's??

Where does the static bit finish_on_completion in the ovm_env.svh used in these multiple run_test()?

My opinion is that the report summary and the copyright message are not included automatically in run_test() in order to preserve backward compatibility for URM users. I think the developers did not want to change the output produced by existing tests (changing logged output can cause havoc with automated regressions). I'm not 100% sure, but that is my guess.

As for the finish_on_completion bit, it does not have anything to do with whether a report summary appears or not. This bit controls whether or not a $finish is executed when run_test() completes *ALL* phases (including the ones after the run phase). Setting this bit would, for example, prevent any code from executing after the call to run_test() completes in the top-level initial block. If you have a call to report_summarize() in your report() method, it will still execute even if finish_on_completion is set (assuming that the phases after run() are actually executed - see the discussion above for workarounds to ensure that this happens).

Regards,
-Kurt

[kurts]

Setting this bit would, for example, prevent any code from executing after the call to run_test() completes in the top-level initial block.

One clarification - run_test forks the call to $finish with a join_none right before it completes. So, if you don't have any blocking statements after your call to run_test(), any code after the call to run_test will execute.

initial begin
...
run_test("testname");
$display("This code will execute regardless of finish_on_completion");
end


inital begin
...
run_test("testname");
#0;
$display("This code will not execute if finish_on_completion is 1");
end

Regards,
-Kurt

[mglasser]

The semantics of run_test() and do_test() are different. Do_test() uses the semantics from AVM where the run() task in the top-level environment controls when the simulation terminates. Run_test(), on the other hand, doesn't handle the run() task in the top-level environment any differently than any other run() task. When using run_test() you must use global_stop_request to shut down the simulatiuon.

The ovm_component::m_do_test_mode bit is set to track which run semantic is desired. If you want do_test() semantics then call do_test() to start your testbench, otherwise call run_test().

By the way, Kurt, your guess is correct: the banner and message summary are not printed to preserve URM backward compatibility.

There really is no need for two ways to start the test running. Do_test() will be deprecated in future versions and run_test will become the preferred way to kick off your test. We'll then add the banner and summary to run_test.

-- Mark

[akhailtash]

I agree that having backward compatibility would be beneficial to many early adopters of AVM/URM and I think should still be there. Except that for people starting to jump on the OVM band wagon makes things more difficult. One does not know which one is the preferred method and which one would be deprecated.

Especially in the examples, I can see that they have not been cleaned up and still use their own methodologies. Say the xbus example has a lot of `message macros that come from URM. And other examples of mixing in their respective starting methodology and not using OVM nomenclature.

It would be nice to somehow separate the compatibility packages and concentrate on cleaning up OVM. It would definitely make the methodology a clean, predictable, easier to master and implement.

Just my opinion,
-- Amal

[dlong]

Hi Mark,

Thanks for the clarification about run_test and do_test. I agree that it makes sense to deprecate do_test and just use run_test. I do however have some concerns about the semantics of run_test in a top-level environment. I believe that requiring the run task to call global_stop_request does not offer sufficient flexibility for a multi-environment testbench. For example, if global_stop_request is called from a top-level environment, how can I be sure that all child environments have completed their required steps or that some other environment doesn't require more time to complete its tasks? Similarly, calling global_stop_request in a child environment might make it difficult to reuse in a different top-level environment.

I think that an hierarchical mechanism that kept track of the state of child environments and automatically stopped the simulation once everyone was finished would be a very useful addition to OVM. I am thinking of something similar to the raise_objection(TEST_DONE) and drop_objection(TEST_DONE) methods found in e. Are there any plans to provide something along these lines?

Regards,
Dave