You say that:

“I don’t believe any type of language is technology in the sense that the
language itself has any special properties that make it better than other
languages.”

But then go on to write:

“Languages that use higher levels of abstraction are more productive to use
precisely because they use a higher level of abstraction.”

These seem contradictory to me: isn’t “more productive” also “better”? In any
case, I think we can agree that some languages are more productive than others,
C vs assembly, e.g.
I would agree that a language isn’t very useful without associated programs to process it; if it was then the language with a syntactic construct
“do-what-I-want” would be the end of the story. However, I do think that language design, e.g. coming up with the right abstractions, can be enormously
powerful. Many people find static scoping in languages like Haskell to be
better than dynamic scoping; it’s subjective, but a “special” property that
can be debated and isn’t really about a “higher level of abstraction”.

If we agree that one language can be more productive than another, then it seems the relevant question to ask is what is the relative benefit I get from a
more productive verification engineer vs a more productive automated algorithm?
I agree that both are important, but I don’t think that one is universally more important than the other. If a new test gen. algorithm covers a few percent more
coverage goals but a new language allows the verification engineer to develop
the original constrained random testbench 25% faster, I’d take the latter.

I would however tend to disagree with this third law. There is an saying that says something like when you solve a bug you’ll be putting another one in that is harder to find. We’ve had this quite a number of times.

The metaphore of black and white balls is a nice one and is valid for bugs that have equal hardness. A bug that is harder to find is like a ball that is harder to catch.
When I look at your graph, I observer that bug hardness doesn’t drop under the value of about 3 once day 225 is passed. So it looks like single event bugs and double event bugs have mostly been found.
Further, I think we have to be careful with how bug hardness is described. In my opinion bug descriptions are localized to the block where they occur. Even if I find a bug during DMA transfer to the memory, I would still write that it happens during memory write bursts forgetting to mention all the conditions that are needed to get the memory doing this kind of burst to that memory. This is where separating IP Verification and Integration verification helps. It is more efficient to verify smaller blocks exhaustively before integrating them in a larger system that is verified too. And that might mean that one needs to split an IP into smaller pieces for verification. This is not often done and it makes certain bugs just harder to find.

To conclude: interesting idea that I’ll thinker about.

The association of the word “law” with scientific laws is not very scientific (irony intended). For example, we call them Maxwell’s equations, not Maxwell’s laws, the simple reason being that each individual law was already named after someone else.

You may have heard of certain laws being called a “square law” or “inverse law”. Newton’s universal law of gravitation is an example of an inverse square law. Put this way, interpreting “law” to mean proven or “assumed to be true” doesn’t make sense. Interpreting “law” to mean “relationship” makes a lot more sense. Amdahl’s law describes a relationship, so it makes sense to call it a law.

It’s interesting to think of why a relationship of quantities is called a law. We usually think of laws as governing behavior – you can’t commit murder or run red lights, for example. Well, scientific laws also govern behavior. Newton’s laws of motion govern how things move, for example.

Moore’s law is an interesting case. It does state a relationship between two quantities, namely chip density and time. However, it doesn’t actually allow you to make compute one quantity from another. The fact that this year is 2009 does not automatically mean you can calculate chip density. Moore’s law says density doubles every 1.5 years, therefore you would need to know what the density was 1.5 years ago in order to predict the density today. Maybe Moore’s rule-of-thumb is more appropriate.

–chris

Grant Martin

2) However, if you are looking for a catch in TTPE, here it is. TTPE has the same problem as formal verification. Namely, to find a formal proof one needs to have a clearly defined search space. (A formal proof is not just a set of operations over some meaningless symbols. Every symbol has to have a definition relating this symbol to the search space). On the contrary, in simulation, one just needs to know how to perform a simulation run. For example, it is quite possible that for some test vectors, simulation is not even possible yet (because some parts of the design are not finished yet). Then one can run simulation over allowable input assignments.

BTW, this is one more answer to your question about separation of formal verification and simulation. One can run simulation even when formal verification is not applicable because the search space is still “under construction”.
3) In theory the answer to the question “Is it possible to produce a proof that is only polynomially larger than the minimum sized proof in polynomial time?” is no. As I mentioned before this is due to the fact that all non-trivial proof systems known so far are most likely non-automatizable. However these results have little bearing on practical applications because they assume that no extra information is given about a particular formula. In reality, every design has tons of extremely useful information that people just do not know how to use. One of the examples of extremely useful information is design high-level structure. When used properly the knowledge of this structure may dramatically reduce the complexity of finding a good proof (the shortest proofs are ones formulated in high-level terms).

Best regards,
Eugene

1) To explain how one can use proofs, I need to formulate two generic problems of testing/simulation (in terms of SAT). Problem number 1 (property checking): given a CNF formula F, generate a set of points P such that F(P)=0 implies that F evaluates to 0 for all points. In this problem, one uses simulation to check if a particular design property holds. Problem number 2 (property preservation): given an unsatisfiable formula F, find a set of points P that detects satisfiable variations F* of F. This problem describes the situation when the property in question holds for the current design and one needs to check if it still holds after a design change (the new design being specified by a CNF formula F*). This change may describe a manufacturing fault or design modification depending on whether the design is implemented in hardware or as a software model.

2) For solving problem 1 (property checking) one can use a proof that F is unsatisfiable under some symplifying assumptions. (I omit explanations here.)

3) Here is how a proof can be used for solving problem number 2 (property preservation). Given a CNF formula F and a proof R, one just builds a proof encoding P={p1,…,pk} of R and uses the input parts of pi as tests. (Here pi is a complete assignment to the variables of F where F specifies a circuit N. So pi can be represented as (ti,yi) where ti is the assignment to the input variables of N and yi is a simulation trace i.e. assignment to internal variables of N.) The main idea here is follows. One can view R as a proof of a complex proposition F (describing the circuit N plus a property). Suppose that N changes into N* due to a technological fault or design correction and this change is described by CNF formula F* (specifying N* plus the same property). Suppose that the property is broken (i.e. F* is satisfiable). Then the proof R is obviously wrong for F*. Since P encodes R there is high probability that there is a point pi of P such that F(pi) != F*(pi). In this case, the input part ti of pi is a test detecting the bug/fault.

4) Here is a more concrete example. Let N be the circuit to be tested. If one generates a “natural” resolution proof R that two copies of N are identical, then a proof encoding P={p1,..,pk} of R contains tests detecting all stuck-at faults (each test ti being the input part of pi of P as described above). Interestingly, a proof encoding of R actually contains a stronger test set (because even a certain subset of P contains tests detecting all stuck-at faults). So not only, TTPE explains why the stuck-at fault model works well in ATPG, but it also shows a way to build stronger test sets.

5) When I say that simulation can benefit from TTPE (even if formal verification turns out to be unscalable) I mean the following. In many cases, one can derive some properties of good test sets without any knowledge of a proof. For example, in the new paper I mentioned before, I show that so called boundary points are good for encoding ANY resolution proof. (A boundary point is a complete assignment p such that all clauses falsified by p share the same variable).

Another approach is to find a proof for a reduced version of the design at hand. Then analyzing this proof one may come up with a better metric to be used for the entire design (e.g. when analyzing the proof one may identify some non-obvious events that have to be tested).

Eugene

1) I am not clear on what benefit the user derives from having a proof of unsatisfiability. There is certainly benefit to knowing that a property is unsatisfiable (no bugs), but I don’t see how the user would use the proof. I think users would want to use the test vectors for regression testing. But, if the design changes, then the proof would change also. Running the old set of vectors on the new design does not mean that all bugs in the new design will be caught. Even on moderate sized designs, the proof could be thousands of vectors. I don’t think anybody is going to spend a lot of time studying that many vectors, especially if the proof changes every time the design changes. Also, I don’t understand your claim that TTPE is useful even if formal verification doesn’t scale. I think the only way to produce a (minimal sized) proof is to use formal verification. I don’t think it is possible to produce minimal sized proofs on large designs since formal verification won’t work on large designs.

2) With respect to the definition of semi-formal verification, I agree that there are different levels of information, but even an implicit representation of what has not been proven is still more than simulation produces. The dividing line for me is zero knowledge vs. more than zero knowledge. An implicit representation is clearly some knowledge and so I think this is a pretty crisp definition.

3) I realized that my argument about what constitutes a complete proof in simulation is not quite right. It is not 2^N for an N-input circuit, but is, in fact, infinite. The issue is that for the proof to be 2^N, we have to make the assumption that the circuit is combinational when it may not be, even if the specification says so. Since, for simulation, I assume zero knowledge, we cannot make that assumption. For example, suppose we are verifying a 2-input AND gate. If we knew it was combinational, the proof would only be four vectors. But, it could have been designed incorrectly such that there is some state in the design and bugs don’t manifest until after more than four vectors have been applied. (This actually happens when designing at the transistor level). For any finite-sized proof, we could insert a time bomb that only triggers once the proof has been exhausted. Therefore, the only way to be sure there is no sequential behavior is to verify an infinite number of vectors.

4) Having said that, pragmatically, detecting whether a circuit has state or not is relatively easy (can be done in polynomial time, I think). Therefore, it is not unreasonable to claim that finite proofs can be generated using simulation provided we allow some amount of formal analysis as long as it is computationally tractable (say, no more than polynomial complexity).

5) The natural follow-on question from that is: are there polynomial formal analyses that can be done to reduce the proof size to less than 2^N? This would be extremely useful, especially if it were exponentially reduced in polynomial time. I am thinking of how NP-hard problems such as traveling salesman can be solved within a finite bound of being optimal in polynomial time. Is it possible to produce a proof that is only polynomially larger than the minimum sized proof in polynomial time?

cheers,

–chris

1) The list of applications of TTPE (Treating Tests as a Proof Encoding) is very long. One of the most important applications is building good metrics. (Currently finding a metric is black magic.) TTPE implies that a formal proof is an “ideal metric”. It contains a complete set of events one needs to test to guarantee that the property in question holds. So, examining formal proofs of design properties and their encodings in terms of tests one actually studies high quality metrics.

2) In a sense, TTPE justifies formal verification once and for all. Before, a verification engineer could say that all formal verification tools are toys that can handle only small design bits. Now, a formal verification person can retort that a formal proof is an ideal metric. Without studying the structure of proofs (and tests encoding those proofs) one can not make progress in the metrics business. Note that this argument works even if formal verification never becomes scalable.
Eugene