368 lines
15 KiB
HTML
368 lines
15 KiB
HTML
|
<!DOCTYPE html>
|
||
|
|
<html>
|
||
|
|
<!-- Created by GNU Texinfo 7.0.1, https://www.gnu.org/software/texinfo/ -->
|
||
|
|
<head>
|
||
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
|
||
|
|
<title>FFmpeg Automated Testing Environment</title>
|
||
|
|
|
||
|
|
<meta name="description" content="FFmpeg Automated Testing Environment">
|
||
|
|
<meta name="keywords" content="FFmpeg Automated Testing Environment">
|
||
|
|
<meta name="resource-type" content="document">
|
||
|
|
<meta name="distribution" content="global">
|
||
|
|
<meta name="Generator" content="makeinfo">
|
||
|
|
<meta name="viewport" content="width=device-width,initial-scale=1">
|
||
|
|
|
||
|
|
<link href="#Top" rel="start" title="Top">
|
||
|
|
<link href="#SEC_Contents" rel="contents" title="Table of Contents">
|
||
|
|
<style type="text/css">
|
||
|
|
<!--
|
||
|
|
div.example {margin-left: 3.2em}
|
||
|
|
ul.mark-bullet {list-style-type: disc}
|
||
|
|
ul.toc-numbered-mark {list-style: none}
|
||
|
|
-->
|
||
|
|
</style>
|
||
|
|
|
||
|
|
|
||
|
|
</head>
|
||
|
|
|
||
|
|
<body lang="en">
|
||
|
|
|
||
|
|
|
||
|
|
<div class="top-level-extent" id="Top">
|
||
|
|
<a class="top" id="SEC_Top"></a>
|
||
|
|
|
||
|
|
<div class="element-contents" id="SEC_Contents">
|
||
|
|
<h2 class="contents-heading">Table of Contents</h2>
|
||
|
|
|
||
|
|
<div class="contents">
|
||
|
|
|
||
|
|
<ul class="toc-numbered-mark">
|
||
|
|
<li><a id="toc-Introduction" href="#Introduction">1 Introduction</a></li>
|
||
|
|
<li><a id="toc-Using-FATE-from-your-FFmpeg-source-directory" href="#Using-FATE-from-your-FFmpeg-source-directory">2 Using FATE from your FFmpeg source directory</a></li>
|
||
|
|
<li><a id="toc-Submitting-the-results-to-the-FFmpeg-result-aggregation-server" href="#Submitting-the-results-to-the-FFmpeg-result-aggregation-server">3 Submitting the results to the FFmpeg result aggregation server</a></li>
|
||
|
|
<li><a id="toc-Uploading-new-samples-to-the-fate-suite" href="#Uploading-new-samples-to-the-fate-suite">4 Uploading new samples to the fate suite</a></li>
|
||
|
|
<li><a id="toc-FATE-makefile-targets-and-variables" href="#FATE-makefile-targets-and-variables">5 FATE makefile targets and variables</a>
|
||
|
|
<ul class="toc-numbered-mark">
|
||
|
|
<li><a id="toc-Makefile-targets" href="#Makefile-targets">5.1 Makefile targets</a></li>
|
||
|
|
<li><a id="toc-Makefile-variables" href="#Makefile-variables">5.2 Makefile variables</a></li>
|
||
|
|
<li><a id="toc-Examples" href="#Examples">5.3 Examples</a></li>
|
||
|
|
</ul></li>
|
||
|
|
</ul>
|
||
|
|
</div>
|
||
|
|
</div>
|
||
|
|
<div class="chapter-level-extent" id="Introduction">
|
||
|
|
<h2 class="chapter">1 Introduction</h2>
|
||
|
|
|
||
|
|
<p>FATE is an extended regression suite on the client-side and a means
|
||
|
|
for results aggregation and presentation on the server-side.
|
||
|
|
</p>
|
||
|
|
<p>The first part of this document explains how you can use FATE from
|
||
|
|
your FFmpeg source directory to test your ffmpeg binary. The second
|
||
|
|
part describes how you can run FATE to submit the results to FFmpeg’s
|
||
|
|
FATE server.
|
||
|
|
</p>
|
||
|
|
<p>In any way you can have a look at the publicly viewable FATE results
|
||
|
|
by visiting this website:
|
||
|
|
</p>
|
||
|
|
<p><a class="url" href="http://fate.ffmpeg.org/">http://fate.ffmpeg.org/</a>
|
||
|
|
</p>
|
||
|
|
<p>This is especially recommended for all people contributing source
|
||
|
|
code to FFmpeg, as it can be seen if some test on some platform broke
|
||
|
|
with their recent contribution. This usually happens on the platforms
|
||
|
|
the developers could not test on.
|
||
|
|
</p>
|
||
|
|
<p>The second part of this document describes how you can run FATE to
|
||
|
|
submit your results to FFmpeg’s FATE server. If you want to submit your
|
||
|
|
results be sure to check that your combination of CPU, OS and compiler
|
||
|
|
is not already listed on the above mentioned website.
|
||
|
|
</p>
|
||
|
|
<p>In the third part you can find a comprehensive listing of FATE makefile
|
||
|
|
targets and variables.
|
||
|
|
</p>
|
||
|
|
|
||
|
|
</div>
|
||
|
|
<div class="chapter-level-extent" id="Using-FATE-from-your-FFmpeg-source-directory">
|
||
|
|
<h2 class="chapter">2 Using FATE from your FFmpeg source directory</h2>
|
||
|
|
|
||
|
|
<p>If you want to run FATE on your machine you need to have the samples
|
||
|
|
in place. You can get the samples via the build target fate-rsync.
|
||
|
|
Use this command from the top-level source directory:
|
||
|
|
</p>
|
||
|
|
<div class="example">
|
||
|
|
<pre class="example-preformatted">make fate-rsync SAMPLES=fate-suite/
|
||
|
|
make fate SAMPLES=fate-suite/
|
||
|
|
</pre></div>
|
||
|
|
|
||
|
|
<p>The above commands set the samples location by passing a makefile
|
||
|
|
variable via command line. It is also possible to set the samples
|
||
|
|
location at source configuration time by invoking configure with
|
||
|
|
<samp class="option">--samples=<path to the samples directory></samp>. Afterwards you can
|
||
|
|
invoke the makefile targets without setting the <var class="var">SAMPLES</var> makefile
|
||
|
|
variable. This is illustrated by the following commands:
|
||
|
|
</p>
|
||
|
|
<div class="example">
|
||
|
|
<pre class="example-preformatted">./configure --samples=fate-suite/
|
||
|
|
make fate-rsync
|
||
|
|
make fate
|
||
|
|
</pre></div>
|
||
|
|
|
||
|
|
<p>Yet another way to tell FATE about the location of the sample
|
||
|
|
directory is by making sure the environment variable FATE_SAMPLES
|
||
|
|
contains the path to your samples directory. This can be achieved
|
||
|
|
by e.g. putting that variable in your shell profile or by setting
|
||
|
|
it in your interactive session.
|
||
|
|
</p>
|
||
|
|
<div class="example">
|
||
|
|
<pre class="example-preformatted">FATE_SAMPLES=fate-suite/ make fate
|
||
|
|
</pre></div>
|
||
|
|
|
||
|
|
<div class="float">
|
||
|
|
<p>Do not put a ’~’ character in the samples path to indicate a home
|
||
|
|
directory. Because of shell nuances, this will cause FATE to fail.
|
||
|
|
</p><div class="type-number-float"><p><strong class="strong">NOTE
|
||
|
|
</strong></p></div></div>
|
||
|
|
<p>To get the complete list of tests, run the command:
|
||
|
|
</p><div class="example">
|
||
|
|
<pre class="example-preformatted">make fate-list
|
||
|
|
</pre></div>
|
||
|
|
|
||
|
|
<p>You can specify a subset of tests to run by specifying the
|
||
|
|
corresponding elements from the list with the <code class="code">fate-</code> prefix,
|
||
|
|
e.g. as in:
|
||
|
|
</p><div class="example">
|
||
|
|
<pre class="example-preformatted">make fate-ffprobe_compact fate-ffprobe_xml
|
||
|
|
</pre></div>
|
||
|
|
|
||
|
|
<p>This makes it easier to run a few tests in case of failure without
|
||
|
|
running the complete test suite.
|
||
|
|
</p>
|
||
|
|
<p>To use a custom wrapper to run the test, pass <samp class="option">--target-exec</samp> to
|
||
|
|
<code class="command">configure</code> or set the <var class="var">TARGET_EXEC</var> Make variable.
|
||
|
|
</p>
|
||
|
|
|
||
|
|
</div>
|
||
|
|
<div class="chapter-level-extent" id="Submitting-the-results-to-the-FFmpeg-result-aggregation-server">
|
||
|
|
<h2 class="chapter">3 Submitting the results to the FFmpeg result aggregation server</h2>
|
||
|
|
|
||
|
|
<p>To submit your results to the server you should run fate through the
|
||
|
|
shell script <samp class="file">tests/fate.sh</samp> from the FFmpeg sources. This script needs
|
||
|
|
to be invoked with a configuration file as its first argument.
|
||
|
|
</p>
|
||
|
|
<div class="example">
|
||
|
|
<pre class="example-preformatted">tests/fate.sh /path/to/fate_config
|
||
|
|
</pre></div>
|
||
|
|
|
||
|
|
<p>A configuration file template with comments describing the individual
|
||
|
|
configuration variables can be found at <samp class="file">doc/fate_config.sh.template</samp>.
|
||
|
|
</p>
|
||
|
|
<p>The mentioned configuration template is also available here:
|
||
|
|
</p><pre class="verbatim">slot= # some unique identifier
|
||
|
|
repo=git://source.ffmpeg.org/ffmpeg.git # the source repository
|
||
|
|
#branch=release/2.6 # the branch to test
|
||
|
|
samples= # path to samples directory
|
||
|
|
workdir= # directory in which to do all the work
|
||
|
|
#fate_recv="ssh -T fate@fate.ffmpeg.org" # command to submit report
|
||
|
|
comment= # optional description
|
||
|
|
build_only= # set to "yes" for a compile-only instance that skips tests
|
||
|
|
ignore_tests=
|
||
|
|
|
||
|
|
# the following are optional and map to configure options
|
||
|
|
arch=
|
||
|
|
cpu=
|
||
|
|
cross_prefix=
|
||
|
|
as=
|
||
|
|
cc=
|
||
|
|
ld=
|
||
|
|
target_os=
|
||
|
|
sysroot=
|
||
|
|
target_exec=
|
||
|
|
target_path=
|
||
|
|
target_samples=
|
||
|
|
extra_cflags=
|
||
|
|
extra_ldflags=
|
||
|
|
extra_libs=
|
||
|
|
extra_conf= # extra configure options not covered above
|
||
|
|
|
||
|
|
#make= # name of GNU make if not 'make'
|
||
|
|
makeopts= # extra options passed to 'make'
|
||
|
|
#makeopts_fate= # extra options passed to 'make' when running tests,
|
||
|
|
# defaulting to makeopts above if this is not set
|
||
|
|
#tar= # command to create a tar archive from its arguments on stdout,
|
||
|
|
# defaults to 'tar c'
|
||
|
|
</pre>
|
||
|
|
<p>Create a configuration that suits your needs, based on the configuration
|
||
|
|
template. The <code class="env">slot</code> configuration variable can be any string that is not
|
||
|
|
yet used, but it is suggested that you name it adhering to the following
|
||
|
|
pattern ‘<samp class="samp"><var class="var">arch</var>-<var class="var">os</var>-<var class="var">compiler</var>-<var class="var">compiler version</var></samp>’. The
|
||
|
|
configuration file itself will be sourced in a shell script, therefore all
|
||
|
|
shell features may be used. This enables you to setup the environment as you
|
||
|
|
need it for your build.
|
||
|
|
</p>
|
||
|
|
<p>For your first test runs the <code class="env">fate_recv</code> variable should be empty or
|
||
|
|
commented out. This will run everything as normal except that it will omit
|
||
|
|
the submission of the results to the server. The following files should be
|
||
|
|
present in $workdir as specified in the configuration file:
|
||
|
|
</p>
|
||
|
|
<ul class="itemize mark-bullet">
|
||
|
|
<li>configure.log
|
||
|
|
</li><li>compile.log
|
||
|
|
</li><li>test.log
|
||
|
|
</li><li>report
|
||
|
|
</li><li>version
|
||
|
|
</li></ul>
|
||
|
|
|
||
|
|
<p>When you have everything working properly you can create an SSH key pair
|
||
|
|
and send the public key to the FATE server administrator who can be contacted
|
||
|
|
at the email address <a class="email" href="mailto:fate-admin@ffmpeg.org">fate-admin@ffmpeg.org</a>.
|
||
|
|
</p>
|
||
|
|
<p>Configure your SSH client to use public key authentication with that key
|
||
|
|
when connecting to the FATE server. Also do not forget to check the identity
|
||
|
|
of the server and to accept its host key. This can usually be achieved by
|
||
|
|
running your SSH client manually and killing it after you accepted the key.
|
||
|
|
The FATE server’s fingerprint is:
|
||
|
|
</p>
|
||
|
|
<dl class="table">
|
||
|
|
<dt>‘<samp class="samp">RSA</samp>’</dt>
|
||
|
|
<dd><p>d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51
|
||
|
|
</p></dd>
|
||
|
|
<dt>‘<samp class="samp">ECDSA</samp>’</dt>
|
||
|
|
<dd><p>76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86
|
||
|
|
</p></dd>
|
||
|
|
</dl>
|
||
|
|
|
||
|
|
<p>If you have problems connecting to the FATE server, it may help to try out
|
||
|
|
the <code class="command">ssh</code> command with one or more <samp class="option">-v</samp> options. You should
|
||
|
|
get detailed output concerning your SSH configuration and the authentication
|
||
|
|
process.
|
||
|
|
</p>
|
||
|
|
<p>The only thing left is to automate the execution of the fate.sh script and
|
||
|
|
the synchronisation of the samples directory.
|
||
|
|
</p>
|
||
|
|
</div>
|
||
|
|
<div class="chapter-level-extent" id="Uploading-new-samples-to-the-fate-suite">
|
||
|
|
<h2 class="chapter">4 Uploading new samples to the fate suite</h2>
|
||
|
|
|
||
|
|
<p>If you need a sample uploaded send a mail to samples-request.
|
||
|
|
</p>
|
||
|
|
<p>This is for developers who have an account on the fate suite server.
|
||
|
|
If you upload new samples, please make sure they are as small as possible,
|
||
|
|
space on each client, network bandwidth and so on benefit from smaller test cases.
|
||
|
|
Also keep in mind older checkouts use existing sample files, that means in
|
||
|
|
practice generally do not replace, remove or overwrite files as it likely would
|
||
|
|
break older checkouts or releases.
|
||
|
|
Also all needed samples for a commit should be uploaded, ideally 24
|
||
|
|
hours, before the push.
|
||
|
|
If you need an account for frequently uploading samples or you wish to help
|
||
|
|
others by doing that send a mail to ffmpeg-devel.
|
||
|
|
</p>
|
||
|
|
<div class="example">
|
||
|
|
<pre class="example-preformatted">#First update your local samples copy:
|
||
|
|
rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite
|
||
|
|
|
||
|
|
#Then do a dry run checking what would be uploaded:
|
||
|
|
rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite
|
||
|
|
|
||
|
|
#Upload the files:
|
||
|
|
rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite
|
||
|
|
</pre></div>
|
||
|
|
|
||
|
|
|
||
|
|
</div>
|
||
|
|
<div class="chapter-level-extent" id="FATE-makefile-targets-and-variables">
|
||
|
|
<h2 class="chapter">5 FATE makefile targets and variables</h2>
|
||
|
|
|
||
|
|
<ul class="mini-toc">
|
||
|
|
<li><a href="#Makefile-targets" accesskey="1">Makefile targets</a></li>
|
||
|
|
<li><a href="#Makefile-variables" accesskey="2">Makefile variables</a></li>
|
||
|
|
<li><a href="#Examples" accesskey="3">Examples</a></li>
|
||
|
|
</ul>
|
||
|
|
<div class="section-level-extent" id="Makefile-targets">
|
||
|
|
<h3 class="section">5.1 Makefile targets</h3>
|
||
|
|
|
||
|
|
<dl class="table">
|
||
|
|
<dt><samp class="option">fate-rsync</samp></dt>
|
||
|
|
<dd><p>Download/synchronize sample files to the configured samples directory.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><samp class="option">fate-list</samp></dt>
|
||
|
|
<dd><p>Will list all fate/regression test targets.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><samp class="option">fate</samp></dt>
|
||
|
|
<dd><p>Run the FATE test suite (requires the fate-suite dataset).
|
||
|
|
</p></dd>
|
||
|
|
</dl>
|
||
|
|
|
||
|
|
</div>
|
||
|
|
<div class="section-level-extent" id="Makefile-variables">
|
||
|
|
<h3 class="section">5.2 Makefile variables</h3>
|
||
|
|
|
||
|
|
<dl class="table">
|
||
|
|
<dt><code class="env">V</code></dt>
|
||
|
|
<dd><p>Verbosity level, can be set to 0, 1 or 2.
|
||
|
|
</p><ul class="itemize mark-bullet">
|
||
|
|
<li>0: show just the test arguments
|
||
|
|
</li><li>1: show just the command used in the test
|
||
|
|
</li><li>2: show everything
|
||
|
|
</li></ul>
|
||
|
|
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">SAMPLES</code></dt>
|
||
|
|
<dd><p>Specify or override the path to the FATE samples at make time, it has a
|
||
|
|
meaning only while running the regression tests.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">THREADS</code></dt>
|
||
|
|
<dd><p>Specify how many threads to use while running regression tests, it is
|
||
|
|
quite useful to detect thread-related regressions.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">THREAD_TYPE</code></dt>
|
||
|
|
<dd><p>Specify which threading strategy test, either ‘<samp class="samp">slice</samp>’ or ‘<samp class="samp">frame</samp>’,
|
||
|
|
by default ‘<samp class="samp">slice+frame</samp>’
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">CPUFLAGS</code></dt>
|
||
|
|
<dd><p>Specify CPU flags.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">TARGET_EXEC</code></dt>
|
||
|
|
<dd><p>Specify or override the wrapper used to run the tests.
|
||
|
|
The <code class="env">TARGET_EXEC</code> option provides a way to run FATE wrapped in
|
||
|
|
<code class="command">valgrind</code>, <code class="command">qemu-user</code> or <code class="command">wine</code> or on remote targets
|
||
|
|
through <code class="command">ssh</code>.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">GEN</code></dt>
|
||
|
|
<dd><p>Set to ‘<samp class="samp">1</samp>’ to generate the missing or mismatched references.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">HWACCEL</code></dt>
|
||
|
|
<dd><p>Specify which hardware acceleration to use while running regression tests,
|
||
|
|
by default ‘<samp class="samp">none</samp>’ is used.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
<dt><code class="env">KEEP</code></dt>
|
||
|
|
<dd><p>Set to ‘<samp class="samp">1</samp>’ to keep temp files generated by fate test(s) when test is successful.
|
||
|
|
Default is ‘<samp class="samp">0</samp>’, which removes these files. Files are always kept when a test
|
||
|
|
fails.
|
||
|
|
</p>
|
||
|
|
</dd>
|
||
|
|
</dl>
|
||
|
|
|
||
|
|
</div>
|
||
|
|
<div class="section-level-extent" id="Examples">
|
||
|
|
<h3 class="section">5.3 Examples</h3>
|
||
|
|
|
||
|
|
<div class="example">
|
||
|
|
<pre class="example-preformatted">make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
|
||
|
|
</pre></div>
|
||
|
|
</div>
|
||
|
|
</div>
|
||
|
|
</div>
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
</body>
|
||
|
|
</html>
|