Playing with the CPU pipeline

This article will show how basic knowledge of a modern CPU’s instruction pipeline can help micro-optimise code at very little cost, using a real world example: the approximation of a trigonometric function. All this without necessarily having to look at lines of assembly code.

The code used for this article is included in the attached file​.

Evaluating polynomials

Who needs polynomials anyway? We’re writing games, not a computer algebra system, after all. But wait! ​Taylor series are an excellent mathematical tool for approximating certain classes of functions. For instance, this is the Taylor series of sin(x) near x = 0:

\[\sin(x) = x - \dfrac{x^3}{3!} + \dfrac{x^5}{5!} - \dfrac{x^7}{7!} + \dfrac{x^9}{9!} - \dfrac{x^{11}}{11!} + \dfrac{x^{13}}{13!} - \dfrac{x^{15}}{15!} + O(x^{16})\]

Truncating the series at the 15th power will compute sin(x) with an absolute error no greater than 1e-11 in the range [-π/2; π/2], and 2e-16 in the range [-π/4; π/4].

However, a better approximation known as the minimax polynomial (probably featured in an upcoming article) will give a maximum absolute error of about 2e-16 on the whole [-π/2; π/2] range: 

static double a0 = +1.0;
static double a1 = -1.666666666666580809419428987894207e-1;
static double a2 = +8.333333333262716094425037738346873e-3;
static double a3 = -1.984126982005911439283646346964929e-4;
static double a4 = +2.755731607338689220657382272783309e-6;
static double a5 = -2.505185130214293595900283001271652e-8;
static double a6 = +1.604729591825977403374012010065495e-10;
static double a7 = -7.364589573262279913270651228486670e-13;

double sin1(double x)
{
    return a0 * x
         + a1 * x * x * x
         + a2 * x * x * x * x * x
         + a3 * x * x * x * x * x * x * x
         + a4 * x * x * x * x * x * x * x * x * x
         + a5 * x * x * x * x * x * x * x * x * x * x * x
         + a6 * x * x * x * x * x * x * x * x * x * x * x * x * x
         + a7 * x * x * x * x * x * x * x * x * x * x * x * x * x * x * x;
}

That is 64 multiplications and 7 additions, though compiler options such as GCC’s -ffast-math will help factor the expression in order to perform fewer operations.

It is possible to help the CPU by noticing that a term such as x^9 can be computed in only one operation if x^2 and x^7 are already known, leading to the following code: 

double sin2(double x)
{
    double ret, y = x, x2 = x * x;
    ret = a0 * y; y *= x2;
    ret += a1 * y; y *= x2;
    ret += a2 * y; y *= x2;
    ret += a3 * y; y *= x2;
    ret += a4 * y; y *= x2;
    ret += a5 * y; y *= x2;
    ret += a6 * y; y *= x2;
    ret += a7 * y;
    return ret;
}

That is now only 16 multiplications and 7 additions. But it is possible to do even better using the ​Horner form of a polynomial evaluation:

\[\sum_{i=0}^n a_i x^i = a_0 + x * (a_1 + x * (a_2 + x * (\dots + x * a_n)))\dots)\]

Leading to the following code: 

double sin3(double x)
{
    double x2 = x * x;
    return x * (a0 + x2 * (a1 + x2 * (a2 + x2 * (a3 + x2 * (a4 + x2 * (a5 + x2 * (a6 + x2 * a7)))))));
}

We are down to 9 multiplications and 7 additions. There is probably no way to be faster, is there? Let’s see…

Timings

Here are the timings in nanoseconds for the above code, compared with the glibc’s sin() function. The test CPU is an Intel® Core™ i7-2620M CPU at 2.70GHz. The functions were compiled using -O3 -ffast-math:

function sin sin1 sin2 sin3
nanoseconds per call 22.518 16.406 16.658 25.276

Wait, what? Our superbly elegant function, performing only 9 multiplications, is actually slower than the 64-multiplication version? Which itself is as fast as the 16-multiplication one? Surely we overlooked something.

That’s right. We ignored the CPU pipeline.

The instruction pipeline

In order to execute an instruction, such as “add A and B into C”, a CPU needs to do at least the following:

  • fetch the instruction (ie. read it from the program’s memory)
  • decode the instruction
  • read the instruction’s operands (ie. A and B)
  • execute the instruction
  • write the result in memory or in registers (in our case, C)

On a modern Intel® CPU, the execution step only accounts for 1/10 or even 1/16 of the total execution time. The idea behind ​pipelining is simple: while executing an instruction, the CPU can often already read the operands for the next one.

But there is a problem with this strategy: if the next instruction depends on the result of the current one, the CPU cannot read the next operands yet. This is called a ​read-after-write hazard, and most usually causes a pipeline stall: the CPU just does nothing until it can carry on.

For the sake of simplicity, imagine the CPU’s pipeline depth is 3. At a given time, it can fetch, execute and finish one instruction:

● instruction is being fetched, executed or finished
○ instruction could start, but needs to wait for the result of a previous instruction

This is how the CPU would execute A = (a + b) * (c + d):

time → total: 7
1 B = a + b ● ● ●
2 C = c + d ● ● ●
3 A = B * C ○ ○ ● ● ●

The c + d operation can be started very early because it does not depend on the result of a + b. This is called ​instruction-level parallelism. However, the final B * C operation needs to wait for all previous instructions to finish.

Since every operation in sin3() depends on the previous one, this is how it would execute that function:

time → total: 48
1 x2 = x * x ● ● ●
2 A = a7 * x2 ○ ○ ● ● ●
3 A += a6 ○ ○ ● ● ●
4 A *= x2 ○ ○ ● ● ●
5 A += a5 ○ ○ ● ● ●
6 A *= x2 ○ ○ ● ● ●
7 A += a4 ○ ○ ● ● ●
8 A *= x2 ○ ○ ● ● ●
9 A += a3 ○ ○ ● ● ●
10 A *= x2 ○ ○ ● ● ●
11 A += a2 ○ ○ ● ● ●
12 A *= x2 ○ ○ ● ● ●
13 A += a1 ○ ○ ● ● ●
14 A *= x2 ○ ○ ● ● ●
15 A += a0 ○ ○ ● ● ●
16 A *= x ○ ○ ● ● ●

These 9 multiplications and 7 additions are done in 48 units of time. No instruction-level parallelism is possible because each instruction needs to wait for the previous one to finish.

The secret behind sin2()’s performance is that the large number of independent operations allows the compiler to reorganise the computation so that the instructions can be scheduled in a much more efficient way. This is roughly how GCC compiled it:

time → total: 30
1 x2 = x * x ● ● ●
2 A = a7 * x2 ○ ○ ● ● ●
3 x3 = x2 * x ● ● ●
4 A += a6 ○ ● ● ●
5 B = a1 * x3 ● ● ●
6 x5 = x3 * x2 ● ● ●
7 A *= x2 ● ● ●
8 C = a2 * x5 ○ ● ● ●
9 B += x ● ● ●
10 x7 = x5 * x2 ● ● ●
11 A += a5 ● ● ●
12 D = a3 * x7 ● ● ●
13 B += C ● ● ●
14 x9 = x7 * x2 ● ● ●
15 B += D ○ ● ● ●
16 E = a4 * x9 ● ● ●
17 x11 = x9 * x2 ● ● ●
18 B += E ○ ● ● ●
19 A *= x11 ● ● ●
20 A += B ○ ○ ● ● ●

These 13 multiplications and 7 additions are executed in 30 units of time instead of 48 for the previous version. The compiler has been rather clever here: the number of ○’s is kept small.

Note that 30 / 48 = 0.625, and the ratio between sin2 and sin3’s timings is 16.658 / 25.276 = 0.659. Reality matches theory pretty well!

Going further

We have seen that increasing the number of operations in order to break dependencies between CPU instructions allowed to help the compiler perform better optimisations taking advantage of the CPU pipeline. But that was at the cost of 40% more multiplications. Maybe there is a way to improve the scheduling without adding so many instructions?

Luckily there are other ways to evaluate a polynomial.

Even-Odd form and similar schemes

Consider our 8th order polynomial:

\[P(x) = a_0 x + a_1 x^3 + a_2 x^5 + a_3 x^7 + a_4 x^9 + a_5 x^{11} + a_6 x^{13} + a_7 x^{15}\]

Separating the odd and even coefficients, it can be rewritten as:

\[P(x) = x \left((a_0 + a_2 x^4 + a_4 x^8 + a_6 x^{12}) + x^2 (a_1 + a_3 x^4 + a_5 x^8 + a_7 x^{12})\right)\]

Which using Horner’s form yields to:

\[P(x) = x \left((a_0 + x^4 (a_2 + x^4 (a_4 + x^4 + a_6))) + x^2 (a_1 + x^4 (a_3 + x^4 (a_5 + x^4 a_7)))\right)\]

This polynomial evaluation scheme is called the Even-Odd scheme. It only has 9 multiplications and 7 additions (only one multiplication more than the optimal case). It results in the following C code: 

double sin4(double x)
{
    double x2 = x * x;
    double x4 = x2 * x2;
    double A = a0 + x4 * (a2 + x4 * (a4 + x4 * a6));
    double B = a1 + x4 * (a3 + x4 * (a5 + x4 * a7));
    return x * (A + x2 * B);
}

And this is the expected scheduling:

time → total: 33
1 x2 = x * x ● ● ●
2 x4 = x2 * x2 ○ ○ ● ● ●
3 B = a7 * x4 ○ ○ ● ● ●
4 A = a6 * x4 ● ● ●
5 B += a5 ○ ● ● ●
6 A += a4 ● ● ●
7 B *= x4 ○ ● ● ●
8 A *= x4 ● ● ●
9 B += a3 ○ ● ● ●
10 A += a2 ● ● ●
11 B *= x4 ○ ● ● ●
12 A *= x4 ● ● ●
13 B += a1 ○ ● ● ●
14 A += a0 ● ● ●
15 B *= x2 ○ ● ● ●
16 A += B ○ ○ ● ● ●
17 A *= x ○ ○ ● ● ●

Still not good enough, but we’re certainly onto something here. Let’s try another decomposition for the polynomial:

\[P(x) = x \left((a_0 + a_3 x^6 + a_6 x^{12}) + x^2 (a_1 + a_4 x^6 + a_7 x^{12}) + x^4(a_2 + a_5 x^6)\right)\]

And using Horner’s form again:

\[P(x) = x \left((a_0 + x^6 (a_3 + x^6 a_6) + x^2 (a_1 + x^6 (a_4 + x^6 a_7)) + x^4(a_2 + x^6 a_5)\right)\]

Resulting in the following code: 

double sin5(double x)
{
    double x2 = x * x;
    double x4 = x2 * x2;
    double x6 = x4 * x2;
    double A = a0 + x6 * (a3 + x6 * a6);
    double B = a1 + x6 * (a4 + x6 * a7);
    double C = a2 + x6 * a5;
    return x * (A + x2 * B + x4 * C);
}

And the following scheduling:

time → total: 31
1 x2 = x * x ● ● ●
2 x4 = x2 * x2 ○ ○ ● ● ●
3 x6 = x4 * x2 ○ ○ ● ● ●
4 B = x6 * a7 ○ ○ ● ● ●
5 A = x6 * a6 ● ● ●
6 C = x6 * a5 ● ● ●
7 B += a4 ● ● ●
8 A += a3 ● ● ●
9 C += a2 ● ● ●
10 B *= x6 ● ● ●
11 A *= x6 ● ● ●
12 C *= x4 ● ● ●
13 B += a1 ● ● ●
14 A += a0 ● ● ●
15 B *= x2 ○ ● ● ●
16 A += C ● ● ●
17 A += B ○ ○ ● ● ●
18 A *= x ○ ○ ● ● ●

One more instruction and two units of time better. That’s slightly better, but still not as good as we would like. One problem is that a lot of time is lost waiting for the value x6 to be ready. We need to find computations to do in the meantime to avoid pipeline stalls.

High-Low form

Instead of splitting the polynomial into its even and odd coefficients, we split it into its high and low coefficients:

\[P(x) = x \left((a_0 + a_1 x^2 + a_2 x^4 + a_3 x^6) + x^8 (a_4 + a_5 x^2 + a_6 x^4 + a_7 x^6)\right)\]

And again using Horner’s form:

\[P(x) = x \left((a_0 + x^2 (a_1 + x^2 (a_2 + x^2 a_3))) + x^8 (a_4 + x^2 (a_5 + x^2 (a_6 + x^2 a_7)))\right)\]

The corresponding code is now: 

double sin6(double x)
{
    double x2 = x * x;
    double x4 = x2 * x2;
    double x8 = x4 * x4;
    double A = a0 + x2 * (a1 + x2 * (a2 + x2 * a3));
    double B = a4 + x2 * (a5 + x2 * (a6 + x2 * a7));
    return x * (A + x8 * B);
}

And the expected scheduling:

time → total: 30
1 x2 = x * x ● ● ●
2 B = x2 * a7 ○ ○ ● ● ●
3 A = x2 * a3 ● ● ●
4 x4 = x2 * x2 ● ● ●
5 B += a6 ● ● ●
6 A += a2 ● ● ●
7 x8 = x4 * x4 ● ● ●
8 B *= x2 ● ● ●
9 A *= x2 ● ● ●
10 B += a5 ○ ● ● ●
11 A += a1 ● ● ●
12 B *= x2 ○ ● ● ●
13 A *= x2 ● ● ●
14 B += a4 ○ ● ● ●
15 A += a0 ● ● ●
16 B *= x8 ○ ● ● ●
17 A += B ○ ○ ● ● ●
18 A *= x ○ ○ ● ● ●

Finally! We now schedule as well as GCC, and with 11 multiplications instead of 13. Still no real performance gain, though.

Pushing the limits

Can we do better? Probably. Remember that each ○ in the above table is a pipeline stall, and any instruction we would insert there would be basically free.

Note the last instruction, A *= x. It causes a stall because it needs to wait for the final value of A, but it would not be necessary if A and B had been multiplied by x beforehands.

Here is a way to do it (bold instructions indicate a new instruction or a modified one):

time → total: 27
1 x2 = x * x ● ● ●
2 B = x2 * a7 ○ ○ ● ● ●
3 A = x2 * a3 ● ● ●
4 x4 = x2 * x2 ● ● ●
5 B += a6 ● ● ●
6 A += a2 ● ● ●
7 x8 = x4 * x4 ● ● ●
8 B *= x2 ● ● ●
9 A *= x2 ● ● ●
10 x3 = x2 * x ● ● ●
11 B += a5 ● ● ●
12 A += a1 ● ● ●
13 C = a0 * x ● ● ●
14 B *= x2 ● ● ●
15 A *= x3 ● ● ●
16 x9 = x8 * x ● ● ●
17 B += a4 ● ● ●
18 A += C ● ● ●
19 B *= x9 ○ ● ● ●
20 A += B ○ ○ ● ● ●

Excellent! Just as many instructions as GCC, but now with fewer pipeline stalls. I don’t know whether this scheduling is optimal for the (incorrect) assumption of a 3-stage pipeline, but it does look pretty good. Also, loading a0, a1 etc. from memory hasn't been covered for the sake of simplicity.

Anyway, we just need to write the code corresponding to this behaviour, and hope the compiler understands what we need: 

double sin7(double x)
{
    double x2 = x * x;
    double x3 = x2 * x;
    double x4 = x2 * x2;
    double x8 = x4 * x4;
    double x9 = x8 * x;
    double A = x3 * (a1 + x2 * (a2 + x2 * a3));
    double B = a4 + x2 * (a5 + x2 * (a6 + x2 * a7));
    double C = a0 * x;
    return A + C + x9 * B;
}

Conclusion

It’s time to check the results! Here they are, for all the functions covered in this article:

function sin sin1 sin2 sin3 sin4 sin5 sin6 sin7
nanoseconds per call 22.518 16.406 16.658 25.276 18.666 18.582 16.366 17.470

Damn. All these efforts to understand and refactor a function, and our best effort actually performs amongst the worst!

What did we miss? Actually, this time, nothing. The problem is that GCC didn't understand what we were trying to say in sin7() and proceeded with its own optimisation ideas. Compiling with -O3 instead of -O3 -ffast-math gives a totally different set of timings:

function sin sin1 sin2 sin3 sin4 sin5 sin6 sin7
nanoseconds per call 22.497 30.250 19.865 25.279 18.587 18.958 16.362 15.891

There. We win eventually!

There is a way to still use -ffast-math yet prevent GCC from trying to be too clever. This might be preferable because we do not want to lose the benefits of -ffast-math in other places. By using an architecture-specific assembly construct, we can mark temporary variables as used, effectively telling GCC that the variable needs to be really computed and not optimised away: 

double sin7(double x)
{
    double x2 = x * x;
    double x3 = x2 * x;
    double x4 = x2 * x2;
    double x8 = x4 * x4;
    double x9 = x8 * x;
#if defined __x86_64__
    __asm__("" : "+x" (x3), "+x" (x9));
#elif defined __powerpc__ || defined __powerpc64__
    __asm__("" : "+f" (x3), "+f" (x9));
#else
    __asm__("" : "+m" (x3), "+m" (x9)); /* Out of luck :-( */
#endif
    double A = x3 * (a1 + x2 * (a2 + x2 * a3));
    double B = a4 + x2 * (a5 + x2 * (a6 + x2 * a7));
    double C = a0 * x;
    return A + C + x9 * B;
}

This works on the x86_64 architecture, where "+x" indicates the SSE registers commonly used for floating point calculations, and on the PowerPC, where "+f" can be used. This approach is not portable and it is not clear what should be used on other platforms. Using "+m" is generic but often means a useless store into memory; however, on x86 it is still a noticeable gain.

And our final results, this time with the full -O3 -ffast-math optimisation flags:

function sin sin1 sin2 sin3 sin4 sin5 sin6 sin7
nanoseconds per call 22.522 16.411 16.663 25.277 18.628 18.588 16.365 15.617

The code used for this article is included in the attached file​.

  • Posted: 2011-09-17 03:58 (Updated: 2014-02-04 01:30)
  • Author: sam
  • Categories: optim code

Attachments (1)

Download all attachments as: .zip

Comments

1. anonymous -- 2012-10-10 19:01

You totally forgot about using SIMD!

2. RuTT -- 2012-10-28 19:08

Very instructive. Thanks!

Note: Mac OS X users, you can use clang++ -stdlib=libc++ poly.cpp to compile the attached file.

3. Jenita -- 2013-04-06 23:39

Heck yeah this is exactly what I ndeeed.

4. anonymous -- 2013-06-12 17:52

Please post more of these gems.. Someone mentioned SIMD (SSE or AVX or whatever version).. wouldn't loading and unloading from that format incur its own hit?

5. anonymous -- 2014-02-03 22:56

Please either use 1/10 or one-tenth, not "one 10th".

6. anonymous -- 2014-02-04 02:23

Can you use -ffast-math and -fno-unsafe-math-optimizations to avoid GCC reordering your code? Regrouping of floating point arithmetic is an "unsafe" optimization since it can change the result (due to different rounding).

7. anonymous -- 2014-02-04 03:56

what is this gcc version? If your processor is capable of, and gcc is recent enough(>4.6, I think) the -ftree-vectorizer option is included into O3, and it should kick in while calculating polynomials.

8. anonymous -- 2014-02-04 04:10

Just tested on a sandy bridge with gcc 4.8, indded it does turn sin7 into simd instructions.

9. anonymous -- 2014-02-04 06:14

OMG It is holly cucumber

10. anonymous -- 2014-02-04 07:05

Nice! Just Superb explanation. Especially for undergraduates.

11. anonymous -- 2014-02-04 11:28

The things I take away from this article are:

1) If you just focus on writing clean and understandable code, the compiler will usually do a pretty good job at optimizing it.

2) If you do decide to hand-optimize your code, be sure to carefully measure the results because it could just as well have the opposite effect.

I've seen a lot of situations where people tried to outsmart a compiler only to make things worse in both performance and maintainability. Micro-optimizations such as these are rarely needed in practice and should always be very carefully considered.

12. anonymous -- 2014-02-04 20:52

Excellent post - thanks!

13. Jonas -- 2014-02-04 21:50

Simply by switching compiler from gcc to clang, I got a factor 2.5 of speedup.

Clang 3.4 

clang++ -march=native -std=c++11 -Ofast poly.cpp && ./a.out
sin: 60.5173 ns
sin1: 6.84736 ns
sin2: 6.83652 ns
sin3: 5.97082 ns
sin4: 5.59059 ns
sin5: 5.80556 ns
sin6: 4.84203 ns
sin7: 5.01364 ns

g++ 4.8.1 

g++ -march=native -std=c++11 -Ofast poly.cpp && ./a.out
sin: 53.8222 ns
sin1: 19.2884 ns
sin2: 12.434 ns
sin3: 20.7135 ns
sin4: 14.9886 ns
sin5: 14.4226 ns
sin6: 13.5111 ns
sin7: 13.0871 ns
14. anonymous -- 2014-02-05 03:39

Also worth noting the processor specified uses simultaneous multithreading, which might have skewed the results a bit.

15. olivier@gautherot.net -- 2014-02-10 13:38

Very informative report but I think it is based on a wrong assumption: it is not simply the pipeline. The i7 (and many more before that!) has a particular type of run-time optimization in the form of a dual pipeline with registers renaming - which is why the execution needs to wait for the previous result to be available.

It would be interesting to evaluate the impact of Hyperthreading (Linux typically divides the dual pipelines to run more processes at the same time, loosing on this particular type of optimization). Also, this should be evaluated on an ARM core, which is very likely not to have this extra boost and probably won't show this discrepancy. I think the comment is interesting but not across the board.

16. embedsys@embedsysweekly.com -- 2014-02-10 22:14

Great post! I have fetured it in the last ​embedded systems weekly issue.
I hope you'll like it

17. anonymous -- 2014-02-11 19:49

Truncating the series at the 15th power will compute sin(x) with an absolute error no greater than 1e-11 in the range [-π/2; π/2], and 2e-16 in the range [-π/4; π/4].

Did you really mean 1e-11, or you wanted to mean 2e-11, since 1e-11 really do not mean anything which is equal to 1 anyway. This might be a typo.

--kongkon

18. sam -- 2014-02-11 22:48

@anonymous: I am using the ​E notation here, so 1e-11 should be understood as 1×10-11.

19. anonymous -- 2014-02-15 23:33

Pure showing-off a useless skill. I prefer to optimize UNTIL it is necessary enough. -- but most of time it turns that Compiler did good enough.

20. Ricnup -- 2014-06-08 22:20

Hello...

21. anonymous -- 2015-03-15 13:16

See also ​https://en.wikipedia.org/wiki/Estrin%27s_scheme

22. Franciswhow -- 2016-10-04 06:14

現金網 ​http://king168.net

120. online -- 2016-12-16 17:15

​http://v9onlinepharm.com/ , and all other CNS stimulants are contraindicated for patients who have undergone CABG.

153. Clintonnib -- 2017-01-28 20:20

Tinedol - an effective remedy for fungal foot odor and itching.

155. CharlesTew -- 2017-07-19 06:41

wh0cd896123 <a href=​http://paroxetine.reisen/>paroxetine hcl 20mg</a> <a href=​http://arimidexformen.pro/>arimidex pills</a> <a href=​http://allopurinol.systems/>allopurinol</a>

156. BennySedly -- 2017-07-19 08:40

wh0cd59278 <a href=​http://effexor.us.org/>Effexor 150mg</a> <a href=​http://buyinderal.us.org/>buy inderal online</a> <a href=​http://diclofenac75mg.us.com/>Diclofenac Sod</a>

158. KennethCeali -- 2017-07-21 05:17

wh0cd516347 <a href=​http://doxycycline100mg.reisen/>doxycycline 100mg</a> <a href=​http://advairdiskus.pro/>advair</a> <a href=​http://buyelimite.store/>cheap elimite</a> <a href=​http://buyvermox.store/>vermox</a> <a href=​http://buyphenergan.shop/>phenergan generic</a> <a href=​http://amoxicillin.work/>amoxicillin prescription</a> <a href=​http://buy-lasix.reisen/>visit website</a> <a href=​http://buysildenafil.reisen/>where to buy sildenafil citrate</a>

160. AlfredKer -- 2017-07-22 16:41

wh0cd708146 <a href=​http://amoxicillin875mg.us.com/>amoxicillin 875 mg</a> <a href=​http://claritingeneric.us.com/>CLARITIN 10MG</a> <a href=​http://amoxicillin.us.org/>purchase amoxicillin</a>

161. CharlesTew -- 2017-07-23 01:00

wh0cd928567 <a href=​http://kytril.world/>kytril</a> <a href=​http://elocon.world/>elocon</a> <a href=​http://vermox.live/>mebendazole tablets</a>

164. CharlesTew -- 2017-07-23 22:30

wh0cd899967 <a href=​http://buysildenafil.reisen/>cheap sildenafil citrate</a> <a href=​http://eloconcreamforsale.pro/>elocon</a> <a href=​http://clomid.tools/>clomid medicine</a> <a href=​http://buy-allopurinol.reisen/>cost of allopurinol</a> <a href=​http://buybupropion.store/>discover more</a> <a href=​http://diflucanoverthecounter.pro/>order diflucan online</a> <a href=​http://prednisone20mg.pro/>prednisone 20mg</a>

165. AlfredKer -- 2017-07-24 04:34

wh0cd312157 <a href=​http://buy-cafergot.reisen/>cafergot</a> <a href=​http://buy-vpxl.shop/>vpxl online</a> <a href=​http://acyclovir800mg.pro/>acyclovir 800 mg tablet</a> <a href=​http://buy-buspar.shop/>buspar</a> <a href=​http://buy-buspar.reisen/>buspirone 5 mg</a> <a href=​http://lipitorgeneric.pro/>lipitor</a> <a href=​http://buyalbuterol.work/>albuterol</a> <a href=​http://tretinoin.fund/>full report</a>

166. AlfredKer -- 2017-07-24 21:42

wh0cd944796 <a href=​http://proscar.reisen/>proscar</a> <a href=​http://buspar.store/>buspar</a> <a href=​http://elimite.work/>generic elimite</a>

167. CharlesTew -- 2017-07-25 02:23

wh0cd34492 <a href=​http://lasix.biz/>Lasix Online</a>

168. KennethCeali -- 2017-07-25 15:50

wh0cd299762 <a href=​http://doxycyclineprice.us.com/>doxycycline</a> <a href=​http://synthroid.us.org/>more information</a>

169. AgustinMAR -- 2017-07-26 10:30

is cbd marijuana legal <a href=​https://cbdpillssale.com/>best cbd capsules reviews</a> cbd pills legal in all 50 states cannabis cbd capsules for sale <a href=​https://cbdcapsulesonline.org/>best cannabis cbd products for fibromyalgia</a> cbd marijuana stocks cannabis cbd oil for pain <a href=​https://cbdpillsonline.com/>side effects of cbd pills</a> cbd cannabidiol cbd pills reviews <a href="​https://cbdpillssale.com/">marijuana cbds and seizures</a> marijuana cbd colorado cbd pills <a href="​https://cbdcapsulesonline.org/">cbd cannabis oil online</a> dosage for cbd capsules cannabidiol cbd oil for stress <a href="​https://cbdpillsonline.com/">cannabidiol cbd pills for sale</a> cbd marijuana strains

170. Michaelsex -- 2017-07-26 12:19

allyauto com <a href="​http://albanyriverrats.com/">auto owners insurance reviews</a> health insurance cost <a href=​http://albanyriverrats.com/>car auto insurance quotes</a> what is the cheapest car insurance

173. TracySueme -- 2017-07-30 01:51

wh0cd43047 <a href=​http://suhagra.us.com/>suhagra</a> <a href=​http://retina24h.us.org/>Retin-A</a>

174. AlfredKer -- 2017-07-30 16:51

wh0cd38328 <a href=​http://anafranilprice.us.com/>buy anafranil online</a>

175. TracySueme -- 2017-08-02 09:40

wh0cd969591 <a href=​http://naprosyn.world/>naprosyn 375 mg tab</a> <a href=​http://septilin.reisen/>septilin</a> <a href=​http://dipyridamole.reisen/>dipyridamole</a> <a href=​http://coreg.reisen/>coreg hypertension</a>

177. AlfredKer -- 2017-08-03 01:01

wh0cd540258 <a href=​http://buyadvair.reisen/>advair</a> <a href=​http://zoloft.news/>zoloft</a> <a href=​http://buytretinoin.reisen/>tretinoin cream</a>

178. KennethCeali -- 2017-08-03 03:12

wh0cd556483 <a href=​http://onlinefurosemide.us.com/>more hints</a>

180. AlfredKer -- 2017-08-04 01:00

wh0cd483000 <a href=​http://genericzoloft.us.com/>continue reading</a>

193. CharlesTew -- 2017-08-06 21:22

wh0cd866602 <a href=​http://zebeta.world/>generic zebeta</a> <a href=​http://florinef.world/>florinef</a> <a href=​http://pilex.reisen/>pilex</a> <a href=​http://clozaril.reisen/>clozaril</a> <a href=​http://lasuna.world/>generic lasuna</a> <a href=​http://cardizem.world/>cardizem 120 mg</a> <a href=​http://diamox.reisen/>diamox</a> <a href=​http://lotensin.world/>lotensin generic</a>

200. CharlesTew -- 2017-08-09 16:11

wh0cd972536 <a href=​http://lasixmedication.us.com/>buy lasix</a> <a href=​http://lipitor.us.com/>lipitor sales</a>

201. anonymous -- 2018-01-16 10:53

This article is showing very well how basic knowledge of a modern CPU’s instruction pipeline can help micro-optimise code at very little cost, using a real world example. Thanks for it and carry on it. To get electrical safety matting Click here ​http://electricalmattingco.co.uk/

202. Best Writers Reviews -- 2018-07-14 12:44

A superior all assignment Help reviews offered by this website with the advantage of online support with high proficiency level based on its latest research and information by professional reviews writers. Wide ranges of subjects are covered with separate writers for each subject. ​http://www.bestwritersreviews.com/a-review-on-allassignmenthelp-com

​http://bestwritersreviews.com/myassignmenthelp-com-reviews

203. anonymous -- 2018-09-24 00:57

Thanks for sharing ​http://www.productosbancarios.net/creditos-rapidos/

204. alicealbert289@gmail.com -- 2018-09-26 07:37

John Arnold Is An Academic Writer Of The Dissertation-Guidance. Who Writes Quality Academic Papers For Students To Help Them In Accomplishing Their Goals. ​https://www.sashelponline.com/

205. structured settlement companies florida -- 2018-09-26 09:28

Things Are Very Open And Intensely Clear Explanation Of Issues. Was Truly Information. Your Website Is Very Beneficial. ​https://structuredsettlementscompanies.xyz/

206. rahil -- 2018-10-05 12:55

Thanks for sharing this information..have shared this link witrh others keep posting such information.. ​https://www.viraltrench.com/top-50-happy-birthday-friend-quote/

207. anonymous -- 2018-10-15 10:38

This is an awesome blog. You provide very useful data. ​https://medium.com/@mahfug50/enlighten-your-mood-by-listening-to-audio-%C7%80-mp3-songs-d47cc2a9683 visit here

208. anonymous -- 2018-10-17 05:58

Thank you for your article. It was very useful for me. I'm happy I found this blog. ​https://onesourcedirectmarketing.com/ email marketing

209. Dave -- 2018-10-24 04:55

Tips to easily login to ​192.168.0.1 as well as other IP addresses

210. anonymous -- 2018-11-09 15:06

This is an awesome blog. You provide very useful data. ​https://mp3beetsongs.wordpress.com/2017/11/30/mankirt-aulaks-badnam-song-is-a-big-hit/ mankirt aulakh’s badnam song

211. anonymous -- 2018-12-01 17:09

Thank you for referring article to me for. You provide very useful data. ​http://whitsundaysdeals.com.au/top-twenty-backpacker-deals/ Whitsundays Deals

212. anonymous -- 2018-12-02 10:49

very informative article. ​https://www.customdeluxeboxes.com/

213. anonymous -- 2018-12-15 11:38

I appreciate you. You provide very useful data. ​https://www.3starhd.me/ download dubbed movies

214. anonymous -- 2018-12-28 11:38

I appreciate you. Great things you've always shared with us. Thanks. Just continue composing this kind of post. ​http://voyance-amour-eternel.com voyante marocaine tres forte

215. anonymous -- 2019-01-11 08:54

I have never read anything like this before. It's completely bold article which using strong idea to express. I really like it. ​Lenny Face

216. AnnaShetty -- 2019-01-29 09:02

I am grateful to have opened this discussion. This question is quite interesting to me. Finally the answer was found ​driving directions

217. anonymous -- 2019-02-09 04:35

Thank you for such a well written article. I am looking forward to sharing your adventures and experiences. <a href="​http://voyance-amour-eternel.com/"> voyance pas chère </a>

218. Leo -- 2019-02-09 14:56

I hope to see more updates from you. Thank you so much for updating ​gmail.com I hope to see more updates from you, it's very helpful for me.

219. anonymous -- 2019-02-12 17:58

Thanks for the incredible blog. I personally like your post. It will help me in great deal. ​http://voyance-retour-amour.com voyance gratuite telephone

220. anonymous -- 2019-02-15 06:10

I am always searching online for articles that can help me and this is one of them. Thanks for starting this. ​http://southviewschool.com/ British curriculum

221. John Nick -- 2019-02-26 08:55

Lucky patcher is a must have application for all the Android users. If you have this application on your device, then you can do a lot with this application. ​https://luckypatcher.news/

222. May -- 2019-03-11 09:37

Thank you so much! This is what I need to find. I look forward to seeing more news from you. ​outlook.com It's so hard to self-learn without this post, very hard.

223. anonymous -- 2019-03-17 07:14

You have befuddling structures to share something like this.Thanks for offer this information. You may check our site other than. ​https://www.ukraynadaegitim.net/zaporijya-teknik-universitesi/ zaporijya

224. mohit4sona@gmail.com -- 2019-03-18 12:35

I am happy to find this post very useful for me, as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. ​https://www.viraltrench.com/best-free-sports-streaming-sites/

225. anonymous -- 2019-03-19 02:48

Appreciative to you for some other edifying site. The spot else may just I get that kind of information written in such a perfect structure. I have an undertaking that I am in a general sense direct running on, and I've been at the post for such data. Thankful you particularly to share these affiliations. ​https://indiantalk.info/apply-for-pmay-online/ pradhan mantri awas yojana online form

226. Lucas -- 2019-03-19 16:58

Thank you for another great article. Where else could anyone get that kind of information in such a perfect way of writing? I have a presentation next week, and I am on the look for such information. ​https://www.hleatherjackets.com/product/avengers-endgame-hoodie/

227. Asad Ali -- 2019-03-24 16:39

What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much. ​https://zartash.pk

228. anonymous -- 2019-04-10 05:33

Your article is valuable and has stunning learning. I respect your undertakings and all the best. It's particularly profitable information. I will sit tight for your next post. ​https://www.azerbaycanegitim.org/azerbaycan-tip-universitesi/ azerbaycan tıp üniversitesi

229. anonymous -- 2019-04-10 07:56

Polynomial math is a part of arithmetic managing images and the principles for controlling those images. In rudimentary polynomial math, those images (today composed as Latin and Greek letters. <a href="​https://prizebondlucky.net/">​https://prizebondlucky.net/</a>

230. anonymous -- 2019-04-15 09:07

Grateful to you for some other illuminating site. The spot else may just I get that sort of data written in such an ideal framework. I have an endeavor that I am essentially directly running on, and I've been at the post for such information. Grateful you especially for sharing these affiliations. ​https://golflaserrangefinderhq.com/best-golf-gps-watch/ this golf rangefinder blog

231. anonymous -- 2019-04-15 11:03

You have made an awesome showing on this article. It's extremely lucid and profoundly smart.<a href="​https://www.lycosgears.com/training-sport/motorcycle-jackets/">Motorcycle Jackets </a>

232. anonymous -- 2019-04-15 11:06

I am impressed by the information that you have on this blog.It shows how well you understand this subject. Bookmarked this page, will come back for more...<a href="​https://www.whatsappsociety.com/">whatsappsociety </a>

233. anonymous -- 2019-04-15 11:09

I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading ..... ​https://creativezone.com.pk

234. anonymous -- 2019-04-18 05:42

This is a magnificent post. Astoundingly illuminating and creative substance. These thinking is an OK technique to empower the information. I like it and help me to advance wonderful. Grateful to you for this short light and surprisingly brilliant data. Everything considered, got a reasonable information. ​https://comparebrokers.co/compare/stock-brokers/ French Stock brokers

235. anonymous -- 2019-04-21 20:06

Thanks for sharing with us watch my blog ​CDHPL

236. Amelie Jones -- 2019-04-28 16:57

Thanks for sharing this effective post it's really useful for me and other users. ​https://www.danezon.com/product/thor-avengers-endgame-cotton-jacket/

237. Gclub -- 2019-05-15 04:34

Thank you for the information that has given us new knowledge that will be used in everyday life. ​https://www.gclub45.com/

238. Gclub-royal1688 -- 2019-05-15 04:35

This website is a very good website that impresses every article. Thank you for the knowledge that has come to us. It's really good. ​https://www.gclub-royal1688.com/

239. pd -- 2019-05-22 01:16

Good website and I love your content. You've got useful information that all of us can learn from. ​tjmaxxcreditcard

241. Giselie jones -- 2019-05-27 12:09

It's incredibly an amazing and key bit of information.I'm in remarkable spirits that you in a general sense offered this confounding information to us <a class="ext-link" href="​https://www.williamjacket.com/product/theodore-decker-the-goldfinch-ansel-elgort-grey-trench-coat/"><span class="icon">​</span>Theodore Decker The Goldfinch Coat</a>

242. anonymous -- 2019-06-09 16:33

Thank you for taking the time to publish this information very useful! ​https://naijaschool.com.ng/waec-gce-results-1st-series/

244. anonymous -- 2019-06-20 08:33

This is definitely an outstanding post and assuming design and pictures here and the design is an excellent way to improve the experience. I like it and help me to advancement very well. Thank you for this brief explanation and very nice information. ​https://nghiviec.com mẫu đơn xin nghỉ việc

245. anonymous -- 2019-06-20 11:02

We endeavor to spare you both time and cash by joining our utilization of material learning, The most astounding quality parts and gear, and our devotion to conveying extraordinary administration.

​https://jfcconstructek.com/articles

246. Tahajafery -- 2019-06-20 11:23

You share the article evaluating polynomials, that is a piece of great information for me, So I want to check your blog again if you add more article about that. ​https://www.vucutguzellik.com/ Here I discuss the cosmetic item

247. Jhon Allen -- 2019-06-20 11:38

This is definitely an outstanding post and assuming design and pictures here and the design is an excellent way to improve the experience. I like it and help me to advancement very well. <a href="​https://www.careerlinecourses.com.au/">​https://www.careerlinecourses.com.au/</a>

Thank you for this brief explanation and very nice information.

248. Ali khan -- 2019-06-20 13:28

Discuss Evaluating polynomials and you share the great article and here I share a useful site about related blog. If you want to use cargo service, ​https://onecargouk.com/air-cargo-services so visit over the site.

249. anonymous -- 2019-06-20 14:26

you are extremely an amazing site administrator. The site stacking pace is dazzling. It kind of feels that you are doing a specific snare. Additionally, You share the article assessing polynomials, that is a bit of extraordinary data for me, So I need to check your blog again in the event that you include more article about that. Online shopping is very important in these days.so if you want to shop online anything than contact us ​https://zonlineshop.com/

250. anonymous -- 2019-06-20 15:32

you are magnificently a befuddling site manager. The site stacking pace is astounding. It kind of feels that you are finishing a specific catch. Besides, You share the article surveying polynomials, that is a bit of unprecedented data for me, So I need to check your blog again in the event that you consolidate more article about that. here if you get a taxi from the airport then visit our site ​https://ukairporttaxistransfers.co.uk/

251. anonymous -- 2019-06-20 20:01

it's packaging, SMS see about the arrived stuff, and moreover readdressing (that suggests changing the objective point or the delegate)The expense of such organizations is resolved freely. ​https://onecargouk.com/sea-cargo-services

252. WalmondZack -- 2019-06-21 13:49

The spot else can also simply I get that form of facts written in such a super framework. I have an undertaking that I am basically directly running on, and I have been on the submit for such statistics. This is surely a first-rate submit and assuming design and photos here and the design is an awesome way to improve the enjoy. ​Write My Assignment

253. anonymous -- 2019-06-22 08:55

I read your article it gives basic of information CPU pipeline but its valuable information. ​http://steininger-online.de

254. anonymous -- 2019-06-22 12:24

I read your article is about general knowledge of instruction CPU pipeline nowadays technology is increased day by day you gave me some important information that I enhance the CPU for drones for kids. ​https://dronerivals.com/top-5-drones-for-kids/

255. Eli Joy -- 2019-06-22 14:10

We provides Dvds <a href="​https://www.dvdshelf.com.au/">dvdshelf</a> in which we provides many informative ideas about CPU pipeline.

256. office.com/setup -- 2019-06-25 07:28

Download office 365 with our assistance [url=​http://aproffice.com/setup]office.com/setup[/url] group without completing any work independent from anyone else.

257. office.com/setup -- 2019-06-25 07:29

Download office 365 with our assistance ​http://aproffice.com/setup group without completing any work independent from anyone else.

258. www.office.com/setup -- 2019-06-25 07:29

Accomplish more with OneDrive and Office ​http://argoffice.com 365 Create cleaned reports, open bits of knowledge, present with lucidity, and work together continuously utilizing Office 365.

259. www.office.com/setup -- 2019-06-25 07:31

The most effective method to login to office.com/setup ? on the off chance that you need to legitimate data about ​http://argoffice.com/setup office.com/setup .Here is our site for your assistance so visit our site you will locate a correct method to work MS office.

260. www.avg.com/retail -- 2019-06-25 07:31

Avg is a well- known name in the field of virus protection. The reason for which this name is common among all end users ​http://avg.com-retail.com

is its free antivirus and malware protection.It not only sounds avg.com/retail free but it provides a lot more with no money. It scans for virus and malware.

261. brother printer setup -- 2019-06-25 07:32

Brother printers are common in both ​http://brother.printersetup.services work and home offices. While setup brother printer the printer out of the box comes with everything you need to install the printer's driver on your computer, not everyone can use the CD-ROM for installation

262. brother printer setup -- 2019-06-25 07:32

Step by step instructions to Install sibling Printer. Numerous individuals have moved toward becoming disappointed ​http://brother.printersupport.support

over an absence of CD to introduce a printer. Presently introducing a printer without a CD does not need to be a troublesome assignment. Simply adhere to a couple of basic guidelines from here and your printer can be introduced.

263. canon printer setup -- 2019-06-25 07:33

Canon printer is a multifunctional printer. Its install automatically on Centrally administered window computer. Canon ​http://canon.printersetup.services

manufactures a wide variety of printers, each with its own particularCanon Printer Wireless Setup features and benefits, such as the ability to scan documents or send faxes.

264. canon printer setup -- 2019-06-25 07:33

Searching for answers for introduce group printer ? Effectively introduce it in your pcs, PCs, work areas, and ​http://canon.printersupport.support

so on with appropriate specialized help from here - introduce ordinance printer.

265. Enter mcafee key code -- 2019-06-25 07:34

Introduce mcafee antivirus in your PC ​http://enterkeycode.com/mcafee

with high class experts and best tech group. Simply ring us and we are prepared to help you till the last moment of establishment - mcafee.com/initiate

266. install norton with key code -- 2019-06-25 07:34

Antivirus is the need of PCs that makes them infection free and we are going to give you full help to get the best ​http://enterkeycode.com/nortonsetup

antivirus introduced in your PCs and PCs. Download norton antivirus bundle with us - norton.com/setup

267. www.office.com/setup -- 2019-06-25 07:35

Outstanding specialized help for ​http://msofficecom-office.com setting up office 365 in your PC or us an approach our number and we are prepared to give you the best help.

268. www.norton.com/setup -- 2019-06-25 07:35

norton web security is broadly utilized ​http://mynortoncom-norton.com

antivirus gives the least demanding to utilize and most intutive assurance for your PC and your mobiles .introduce it and disregard viruses,spyware,root-units, hackers.for more subtleties visit:

269. www.mcafee.com/activate -- 2019-06-25 07:35

Introduce mcafee antivirus in your PC ​http://mcafeecom.us with high class experts and best tech group. Simply ring us and we are prepared to help you till the last moment of establishment - mcafee.com/initiate

270. www.mcafee.com/activate -- 2019-06-25 07:36

mcafee.com/initiate have the total ​http://mcafeecom.us/activate arrangement of highlights which can ensure your computerized on the web and disconnected existence of the processing gadgets, and it causes you secure it as well as it can keep up the solidness of your PC.

271. www.norton.com/setup -- 2019-06-25 07:36

Norton is the most reliable antivirus ​http://my-nortoncom.us and we are giving you the full support to get it installed in your laptops and computers - norton.com/setup

272. www.norton.com/setup -- 2019-06-25 07:37

Download norton antivirus to make your ​http://my-nortoncom.us/setup PC infection free with the best help and tech group. Don't hesitate to get in touch with us

273. Farhanfida -- 2019-06-25 08:21

You share the article about the study, that is a piece of great information for me. And here I also share related data about health, ​https://www.plixlife.com/product/strength/ article is basically focused on health. Because health is the most important thing in our life. And health is also the main purpose of the study.

Add New Comment