The models each simulator had to go through were models from biomodels.net. We used the ninth release (25th September 2007).
Each simulator had to simulate the model for 10 seconds, delivering 1000 simulation points.
In this comparison we only compared the concentration data returned for floating
species.
Results summary
last update - 2008-03-28 - ByoDyn results are added
| Simulator |
passed* |
no difference |
small difference |
significant difference |
| BioUML v.0.7.8 beta |
150 |
136 |
129 |
4 |
8 |
10 |
13 |
| ByoDyn |
149 |
134 |
129 |
2 |
7 |
13 |
13 |
| SBWOdeSolver |
143 |
134 |
126 |
7 |
13 |
2 |
4 |
| RoadRunner |
137 |
115 |
107 |
4 |
9 |
18 |
21 |
| Copasi |
133 |
120 |
106 |
5 |
10 |
8 |
17 |
| MathSBML |
134 |
115 |
104 |
8 |
16 |
11 |
14 |
| SBToolbox2 |
106 |
54 |
50 |
38 |
36 |
14 |
20 |
| Oscill8 Core |
106 |
69 |
57 |
2 |
4 |
35 |
45 |
| JSim |
88 |
75 |
73 |
0 |
0 |
13 |
15 |
| Jarnac |
72 |
66 |
65 |
0 |
1 |
6 |
6 |
| VCell |
62 |
54 |
52 |
2 |
2 |
6 |
8 |
| CVODE |
9 |
8 |
7 |
1 |
2 |
0 |
0 |
Implementation note:
* To pass a test a simulator should provide at least values for one variable
excluding time. So if the CSV file contains only time values the test is not
passed.
Details
Simulator data (CSV files):
- CSV files for Copasi, CVODE, RoadRunner, Jarnac, JSim, Oscill8 Core,
SBWOdeSolver were obtained by e-mail from Frank Bergmann at 13.11.2007
- CSV files for MathSBML and SBToolbox2 were downloaded from web site: http://www.sys-bio.org/sbwWiki/compare
Comparison criteria
Passed – CSV file was generated by simulator and the file contains values
at least for one variabel excluding time.
interval criteria
- no difference - 0.999 * min < x < 1.001 * max or
x < ZERO and
max < ZERO
- small difference – 0.5 * min < x < 1.5 * max
- significant
difference - otherwise
median criteria
- no difference - Abs( (x-median)/median ) < 0.01 or
x < ZERO and median < ZERO
- small difference – Abs( (x- median)/median ) < 0.5
- significant difference – otherwise
x – variable value provided by compared simulator
min, max, median –
calculated from values provided by other simulators with which the specified
simulator is being compared.
ZERO values (it also can be considered as relative error) were specified explicitly for following models:
| Model |
ZERO |
| BIOMD0000000002 |
1e-12 |
| BIOMD0000000005 |
1e-10 |
| BIOMD0000000007 |
1e-10 |
| BIOMD0000000009 |
1e-10 |
| BIOMD0000000009 |
1e-10 |
| BIOMD0000000011 |
1e-10 |
| BIOMD0000000013 |
1e-10 |
| BIOMD0000000014 |
1e-12 |
| BIOMD0000000016 |
1e-10 |
| BIOMD0000000017 |
1e-10 |
| BIOMD0000000018 |
1e-20 |
| BIOMD0000000025 |
1e-25 |
| BIOMD0000000028 |
1e-10 |
| BIOMD0000000032 |
1e-10 |
| BIOMD0000000033 |
1e-12 |
| BIOMD0000000038 |
1e-10 |
| BIOMD0000000040 |
1e-10 |
| BIOMD0000000041 |
1e-12 |
| BIOMD0000000046 |
1e-15 |
| BIOMD0000000048 |
1e-14 |
| BIOMD0000000049 |
1e-13 |
| BIOMD0000000051 |
1e-10 |
| BIOMD0000000061 |
1e-10 |
| BIOMD0000000064 |
1e-15 |
| BIOMD0000000069 |
1e-12 |
| BIOMD0000000070 |
1e-10 |
| BIOMD0000000071 |
1e-10 |
| BIOMD0000000072 |
1e-12 |
| BIOMD0000000075 |
1e-10 |
| BIOMD0000000080 |
1e-10 |
| BIOMD0000000081 |
1e-10 |
| BIOMD0000000082 |
1e-13 |
| BIOMD0000000085 |
1e-14 |
| BIOMD0000000086 |
1e-15 |
| BIOMD0000000087 |
1e-11 |
| BIOMD0000000088 |
1e-15 |
| BIOMD0000000091 |
1e-10 |
| BIOMD0000000093 |
1e-11 |
| BIOMD0000000094 |
1e-12 |
| BIOMD0000000103 |
1e-14 |
| BIOMD0000000104 |
1e-10 |
| BIOMD0000000105 |
1e-11 |
| BIOMD0000000106 |
1e-15 |
| BIOMD0000000108 |
1e-10 |
| BIOMD0000000109 |
1e-9 |
| BIOMD0000000111 |
1e-10 |
| BIOMD0000000112 |
1e-10 |
| BIOMD0000000122 |
1e-10 |
| BIOMD0000000123 |
1e-10 |
| BIOMD0000000125 |
1e-10 |
| BIOMD0000000126 |
1e-12 |
| BIOMD0000000137 |
1e-10 |
| BIOMD0000000139 |
1e-12 |
| BIOMD0000000140 |
1e-13 |
| BIOMD0000000143 |
1e-10 |
| BIOMD0000000146 |
1e-09 |
| BIOMD0000000148 |
1e-10 |
| BIOMD0000000150 |
1e-10 |
| BIOMD0000000151 |
1e-12 |
|
