716670 - be able to send graph server results to a file instead of to a server

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Jeff Hammel]

Currently, Talos sends its results directly to GraphServer.  As part
of https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise , we
will want to introspect this data locally.  Capability should be added
to dump the results to a file, possibly using a file:// url vs the
current `results_server, results_link` syntax

Comment 1

[Jeff Hammel]

Attachment: what we send to graphserver

from

(talos)│python run_tests.py 20120109_1448_config.yml --results-url file:///home/jhammel/output.txt

Comment 2

[Jeff Hammel]

Attachment: use generalized URL for graph output

Instead of splitting via results_server and results_link, this patch uses a single url like `results_url = 'http://foo.com/bar'` or `results_url = 'file:///foo.txt'`.  The host and path info pieces are parsed out as required (as well as the scheme).

I also took a first stab at overhauling configuration for talos.  The work is done in earnest as part of bug 704654, but I figure this is a step in the right direction. I also support the old --resultsServer and --resultsLink flags in PerfConfigurator.py as otherwise buildbot will break!  But they are marked as deprecated and the new --resultsURL is prefered.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 587204 [details] [diff] [review]
use generalized URL for graph output

Review of attachment 587204 [details] [diff] [review]:
-----------------------------------------------------------------

hey, this is great.  Has this run through try server?  and a local mobile test by chance?

::: talos/run_tests.py
@@ +587,5 @@
>    parser.add_option('--amo', dest='amo',
>                      action='store_true', default=False,
>                      help="set AMO")
> +  parser.add_option('--results-url', dest='results_url',
> +                    help="url to put graph results to")

why is this here?  Shouldn't this be in PerfConfigurator?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

the what we sent to the graph server is a ts run, not very interesting for a pageloader test.  Tsvg or Tdhtml would be good output to have.

Comment 5

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #3)
> Comment on attachment 587204 [details] [diff] [review]
> use generalized URL for graph output
> 
> Review of attachment 587204 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> hey, this is great.  Has this run through try server?  and a local mobile
> test by chance?

Not yet.  I wanted to see if in general this was something we wanted to take.  I'll run it through try after i unbitrot it (again).

> 
> ::: talos/run_tests.py
> @@ +587,5 @@
> >    parser.add_option('--amo', dest='amo',
> >                      action='store_true', default=False,
> >                      help="set AMO")
> > +  parser.add_option('--results-url', dest='results_url',
> > +                    help="url to put graph results to")
> 
> why is this here?  Shouldn't this be in PerfConfigurator?

In general I would like to mirror the command line handling between PerfConfigurator and run_tests.py. I am not sure why --amo and --screen are here either.  I can take this as part of this patch, if desired.

Comment 6

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #5)
> (In reply to Joel Maher (:jmaher) from comment #3)
<snip/>
> > ::: talos/run_tests.py
> > @@ +587,5 @@
> > >    parser.add_option('--amo', dest='amo',
> > >                      action='store_true', default=False,
> > >                      help="set AMO")
> > > +  parser.add_option('--results-url', dest='results_url',
> > > +                    help="url to put graph results to")
> > 
> > why is this here?  Shouldn't this be in PerfConfigurator?
> 
> In general I would like to mirror the command line handling between
> PerfConfigurator and run_tests.py. I am not sure why --amo and --screen are
> here either.  I can take this as part of this patch, if desired.

As discussed on IRC, I will remove this for now and file a follow-up bug.

Comment 7

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #6)

> As discussed on IRC, I will remove this for now and file a follow-up bug.

bug 716921

Comment 8

[Jeff Hammel]

Attachment: unbitrot and fix some things

will test on try and locally; any particular mobile tests to try?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 587819 [details] [diff] [review]
unbitrot and fix some things

Review of attachment 587819 [details] [diff] [review]:
-----------------------------------------------------------------

this is looking great

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

hmm, further testing of this patch yields a conflict with --develop, here is a diff to fix that:

diff -r b6e86da2f61f talos/PerfConfigurator.py
--- a/talos/PerfConfigurator.py	Wed Jan 11 22:00:19 2012 -0500
+++ b/talos/PerfConfigurator.py	Wed Jan 11 22:05:05 2012 -0500
@@ -476,21 +476,19 @@
         if not options.activeTests:
             raise Configuration("Active tests should be declared explicitly. Nothing declared with --activeTests.")
 
-        if options.develop == True:
-            if options.resultsServer == '':
-                options.resultsServer = ' '
-            if options.resultsLink == '':
-                options.resultsLink = ' '
-
-            if options.webServer == '':
-              options.webServer = "localhost:%s" % (findOpenPort('127.0.0.1'))
-
         # if resultsServer and resultsLinks are given replace resultsURL from there
         if options.resultsServer and options.resultsLink:
             if options.resultsURL:
                 raise Configuration("Can't use resultsServer/resultsLink and resultsURL; use resultsURL instead")
             options.resultsURL = 'http://%s%s' % (options.resultsServer, options.resultsLink)
 
+        if options.develop == True:
+            if options.resultsURL == '':
+                options.resultsURL = ' '
+
+            if options.webServer == '':
+              options.webServer = "localhost:%s" % (findOpenPort('127.0.0.1'))
+
         # XXX delete deprecated values
         del options.resultsServer
         del options.resultsLink

Comment 11

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #10)
> hmm, further testing of this patch yields a conflict with --develop, here is
> a diff to fix that:
> 
> diff -r b6e86da2f61f talos/PerfConfigurator.py
> --- a/talos/PerfConfigurator.py	Wed Jan 11 22:00:19 2012 -0500
> +++ b/talos/PerfConfigurator.py	Wed Jan 11 22:05:05 2012 -0500
> @@ -476,21 +476,19 @@
>          if not options.activeTests:
>              raise Configuration("Active tests should be declared
> explicitly. Nothing declared with --activeTests.")
>  
> -        if options.develop == True:
> -            if options.resultsServer == '':
> -                options.resultsServer = ' '
> -            if options.resultsLink == '':
> -                options.resultsLink = ' '
> -
> -            if options.webServer == '':
> -              options.webServer = "localhost:%s" %
> (findOpenPort('127.0.0.1'))
> -
>          # if resultsServer and resultsLinks are given replace resultsURL
> from there
>          if options.resultsServer and options.resultsLink:
>              if options.resultsURL:
>                  raise Configuration("Can't use resultsServer/resultsLink
> and resultsURL; use resultsURL instead")
>              options.resultsURL = 'http://%s%s' % (options.resultsServer,
> options.resultsLink)
>  
> +        if options.develop == True:
> +            if options.resultsURL == '':
> +                options.resultsURL = ' '
> +
> +            if options.webServer == '':
> +              options.webServer = "localhost:%s" %
> (findOpenPort('127.0.0.1'))
> +
>          # XXX delete deprecated values
>          del options.resultsServer
>          del options.resultsLink
To be honest, I've never understood why we replace '' with ' ' in develop mode.  Can you explain this?  (a comment would also be nice)

Comment 12

[Jeff Hammel]

results on try: https://tbpl.mozilla.org/?tree=Try&rev=eebef9246a78 

Looking good

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

I had added that in a while ago so I could run without errors on my local box while it tried to upload to the servers.  Basically that was a way to ensure it wasn't blank.  I can't remember why "" was insufficient at the time.

Comment 14

[Mozilla RelEng Bot]

Try run for eebef9246a78 is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=eebef9246a78
Results (out of 81 total builds):
    success: 79
    warnings: 1
    failure: 1
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-eebef9246a78
 Timed out after 06 hours without completing.

Comment 15

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #13)
> I had added that in a while ago so I could run without errors on my local
> box while it tried to upload to the servers.  Basically that was a way to
> ensure it wasn't blank.  I can't remember why "" was insufficient at the
> time.

I'll file a follow-up bug about tracking this down.  We shouldn't really have code that we don't understand (especially code that seems unnecessary)

Comment 16

[Jeff Hammel]

pushed: http://hg.mozilla.org/build/talos/rev/2b0122185abd

Comment 17

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #15)
> (In reply to Joel Maher (:jmaher) from comment #13)
> > I had added that in a while ago so I could run without errors on my local
> > box while it tried to upload to the servers.  Basically that was a way to
> > ensure it wasn't blank.  I can't remember why "" was insufficient at the
> > time.
> 
> I'll file a follow-up bug about tracking this down.  We shouldn't really
> have code that we don't understand (especially code that seems unnecessary)

see bug 717693